#018 - Automation for Civil/Structural Engineers | Selenium
#018 - Automation for Civil/Structural Engineers | Selenium
Dive into the world of Selenium, where automation meets engineering, and discover how Python can transform your workflow into streamlined, innovative solutions.
This post is part of the Flocode ‘Open Source Series’, where we talk about the best open-source tools for engineers.
Open-source code is freely available for anyone to modify and redistribute and represents a new paradigm of transparency and collaborative innovation for engineering. Open-source tools are the core of the Flocode philosophy.
Other entries to this series will include:
Introduction
Automation is becoming increasingly commonplace in civil and structural engineering and can offer significant efficiency improvements.
It comes in different flavours, with most people assuming it is a product of company-wide systems/infrastructure or the commercial software you use regularly. What if you could automate aspects of your workflow specific to your individual use cases?
This guide introduces Selenium to beginners.
Selenium is a tool designed for automating web browsers. For engineers interested in leveraging Python, Selenium provides a practical way to automate routine web-based tasks, such as data extraction/entry and form submission.
Integrating Selenium's automation capabilities with other Python libraries allows you to establish a comprehensive automation framework that interacts seamlessly with your core operating system and commercial engineering software programs. With Selenium, you can extend this powerful automation from your desktop to the web.
This allows engineers to streamline parts of their workflow and focus energy on a project’s more complex and creative aspects.
Flocode Difficulty Scale for Selenium: 6/10
Mastering Selenium involves a basic understanding of Python and some essential web development concepts, particularly an understanding of HTML components and the structure of web pages and data presentations online. This foundational knowledge is crucial to effectively utilize Selenium for automating web-based tasks. This article walks you through the fundamentals without any heavy lifting.
What is Selenium?
Selenium is a Python library tool that enables users to mimic human interaction with various websites or web elements. Selenium is designed to facilitate the testing of web applications by automating user actions. This capability is particularly beneficial in ensuring that web applications function as intended across different browsers and platforms. Selenium is not a singular tool but a suite comprising Selenium WebDriver, Selenium IDE, and Selenium Grid.
Selenium WebDriver is a collection of language-specific bindings to drive a browser, allowing you to control a browser programmatically. This is the primary use case for engineers. 👍
Selenium IDE is a browser extension that provides a user-friendly interface for creating automated tests. It is ideal for beginners due to its record-and-playback features (think of recording a macro in Excel). However, features are limited to browsers only. 👎
Selenium Grid allows for running tests on different machines and browsers concurrently, facilitating cross-browser testing. This is a more advanced capability and a discussion for another day, but it is useful for running multiple simultaneous threads of the same analysis type.
Integration with Python
Engineers can use Python to control Selenium. Python's readability and straightforward syntax lower the barrier to entry, making it easier for non-programmers to start automating tasks. Python also bridges the gap between web browser automation and local file/data/software automation.
Web-Based Applications in Engineering
Web-based applications are becoming a standard part of engineering as more databases, services, and software move online to subscription-based services. Online databases for material properties, cloud-based design and analysis tools, and collaborative project management platforms are just a few examples.
Why is Selenium Useful for Engineers?
You could write a Python script using Selenium to open a web page automatically, fill out a form, or retrieve data from a site. This is done by specifying elements on the web page (like buttons or text fields) in the script and defining actions to be performed on these elements.
You can automate an entire series of steps or loops to run through a list of beam sizes, coordinates, addresses, prices, you name it, and carry out subsequent calculations or analyses based on different logic flows built into your process.
Combining all of these steps together is where the practicality becomes more apparent.
Getting Started
To start, engineers install the Selenium package in Python, typically done with a simple command like pip install selenium. Once installed, the core component of Selenium is the WebDriver. This WebDriver acts as a controller for a web browser, allowing scripts to perform actions as if a user were manually interacting with the browser. As always, see the official Selenium Docs for more guidance on getting up and running.
If you are just starting out with Python and this process frustrates you to the point of night sweats, that’s perfectly normal. Flocode’s courses are coming soon; you can sign up on the waitlist.
Enhanced Data Handling
Automating data entry and extraction with Selenium offers a precise and reliable alternative to manual processing. Manual data handling, especially in large volumes, is prone to human error and inconsistency. Selenium scripts, once correctly written and tested, perform tasks with a high degree of accuracy. They follow the same steps every time they are run, eliminating the variability introduced by human input.
While errors and bugs can occur in code, they are often easier to identify and fix than errors in manually entered data. This reliability is particularly crucial in engineering, where data accuracy is what we rely upon to create value.
In manual data entry, the quality and attention to detail can diminish over time due to factors like fatigue or loss of focus. Automated processes, in contrast, do not suffer from this decline. A Selenium script that extracts data from a website will do so with the same level of accuracy on the hundredth run as it did on the first.
By relying on automation for routine tasks, engineers can allocate more attention to complex problem-solving and innovative design work, where human expertise is irreplaceable. Get your brain back. Give yourself the gift of your own brain.
Interfacing with Other Tools
Selenium can interface with various software tools that are staples in the civil and structural engineering fields. While Selenium specializes in automating web browsers, its integration with Python opens up possibilities for interacting with a wide array of engineering applications. Below are some examples of how Selenium can be used in conjunction with these tools:
Excel: Engineers can use Python libraries like
openpyxlorpandasalongside Selenium to manipulate Excel files. Data extracted from web applications using Selenium can be directly fed into Excel spreadsheets for further analysis or reporting.Mathcad, CAD, Civil3D, Revit: While Selenium doesn’t directly interact with these software packages, Python scripts can handle data preparation or post-processing tasks. For instance, Selenium can automate data collection from web sources, which Python can then format into a compatible structure for import into these CAD tools.
SAP2000, ANSYS, MIDAS, SFRAME, CSI Bridge: Similar to CAD tools, Selenium can assist in automating the pre-processing or post-processing phases of analysis in these various structural analysis and design software. Data collected via Selenium can be prepared using Python for input into these programs, or results from these tools can be extracted and processed further. Examples include steel beam properties or mechanical properties for a specific product.
QGIS, HEC-RAS: For GIS and hydrological modelling tools like QGIS and HEC-RAS, Selenium can be used to gather geographical or hydrological data from various online sources. Python can then format this data for use in these applications.
GEOSTUDIO: In geotechnical engineering software like GEOSTUDIO, data regarding soil properties, geotechnical parameters, or design specifications can be collected through Selenium and processed for input.
Python Libraries: Selenium's integration with Python makes it a powerful ally when used in conjunction with Python libraries for data analysis, machine learning, or visualization, such as NumPy, SciPy, Matplotlib, or Scikit-learn.
It’s important to note that while Selenium is not a direct bridge to these applications, it is a crucial component in a larger automation workflow.
By automating data collection and initial processing, Selenium, in tandem with Python, significantly reduces the manual effort required in managing data across various engineering software platforms, thereby enhancing overall productivity and efficiency in engineering projects.
How Do I Start?
Before diving into Selenium, it's important to master some fundamental concepts in Python and Web Development. While it's always possible to learn through determination and persistence, grasping these basics will pave a smoother path to using Selenium effectively. Here are some essential precursors for a more straightforward journey with Selenium:
Basic Python Knowledge: Understanding of Python syntax and programming concepts.
Web Data Access: Familiarity with accessing data from websites, including navigating through web pages and understanding URL structures.
Understanding HTML and CSS: Knowledge of HTML and CSS to identify and interact with web elements.
XPath or CSS Selectors: Ability to use XPath or CSS selectors to locate web elements.
Web Browser Operations: Basic understanding of web browsers' functions, including session management and page navigation.
Data Flow Understanding: Clarity on the inline data flow and the steps involved in data extraction from web pages.
Debugging Skills: Ability to troubleshoot and debug Python and Selenium code.
Environment Setup: Knowledge of setting up a Python environment and installing packages like Selenium WebDriver.
Basic Automation Concepts: Familiarity with automation principles, including repetitive task identification and automation scripting.
I understand this sounds like a lot for beginners, and it is, but you need to understand the broader context of how systems work before you can automate things.
Examples
Below are two examples to show the power of Selenium. The easiest way to start is by using the Selenium Chrome Extension to help you record your steps.
The Chrome extension works like a macro recorder for Excel where you can press record, and your mouse movements and keyboard inputs are recorded until you stop. This is a simple way to sequence a series of steps online, like filling out forms or automating any multi-step process.
I use Brave browser, which also works with Chrome extensions.
A more nuanced version of web automation, in Example 2, uses the Selenium Web Driver and specific Python scripts and workflows to automate sequences for online and offline work. Often, you'll need to automate a combination of web inputs, web outputs, local calculations, local database queries, etc., to create useful workflows. It takes more legwork, but it's an excellent way to automate repetitive or iterative processes.
Example 1: Selenium IDE - Automated Download of AISC Shapes Database
Using the Selenium Chrome Extension, I recorded a series of simple steps described below. You can download this file and open it using your browser. Don’t get hung up on the JavaScript code; this is how Selenium records actions.
Here's a simplified summary of the automated sequence:
Opening Google: The script starts by launching Google's homepage.
Activating Search Bar: It then focuses on the search bar, preparing it for input.
Search Query Entry: The script types "aisc shapes database" into the search bar. This is a query for a database of American Institute of Steel Construction (AISC) shapes.
Selecting Search Suggestion: After typing, the script selects a suggested search item from a dropdown list, refining the search.
Accessing Search Results: It clicks on the first search result related to the AISC shapes database.
Downloading a File: The script then finds and clicks a download link for the AISC Shapes Database (version 16.0) spreadsheet.
Handling Browser Windows: The script includes commands for managing browser windows.
Closing the Browser: Finally, the script closes the browser window or tab once the actions are completed.
This is an arbitrary example, but you can use your imagination to determine what’s practical.
The Selenium IDE is limited to browser-only automation. While you can create some limited JavaScript functions, they will only relate to online browsing activity, and you can not access local files on your machine (i.e., programs or Python files/functions).
Consequently, I recommend jumping directly into Selenium Web Driver, which we will cover next.
Example 2: Selenium Web Driver - Web Scraping Climate Data for Thermal Expansion Calculations
This example demonstrates the application of Selenium for automating data retrieval and processing tasks.
If you're comfortable with Python, this will be relatively straightforward. If not, don't worry. This is what Flocode is all about. We show you the ropes, and you can tackle this kind of thing when you're ready.
It covers three main aspects:
Extracting Data from AISC Shapes Database: The code extracts the cross-sectional area of a specific AISC shape (e.g., W36X318) from the AISC Shapes Database, which is provided in an Excel file. It utilizes Pandas to read the data and extract the required information, providing the cross-sectional area of the steel section in square inches.
Function to extract the cross-sectional area from the AISC Steel Shapes Database Web Scraping with Selenium: This section showcases how Selenium can be employed for web scraping tasks. It specifically targets the US climate data website to extract maximum and minimum monthly average temperatures from tables embedded in the web page for a location (Anchorage, Alaska). Selenium is chosen due to its ability to interact with web elements and extract data from dynamically loaded pages, which traditional HTTP requests cannot handle.
Thermal Expansion Analysis in Python: The code then analyzes the thermal expansion of a steel member subjected to these potential temperature changes. It uses the coefficient of thermal expansion for A572 Grade 50 steel, previously extracted temperature data, and a range of member lengths to calculate and plot thermal expansion versus member length.
If you want to dig into this specific example and the code in more detail, you can fork the GitHub Repo and modify it to fit your needs.
I’ve added plenty of notes in the Jupyter Notebook to facilitate self-directed learning for those interested.
Closing
Selenium is your gateway to the web. It connects your desktop design environment with any website or online data source. The possibilities are limited only by your imagination.
We are just getting started on the automation topic; there’s much to learn in this space. As we progress the series, you can choose the tools that fit your use cases and workflows.
If you have thoughts, ideas or questions, leave them below in the comments. And if you haven’t already, subscribe to this newsletter.
The Flocode community is growing; I appreciate your time and attention.
I’ve been receiving queries about Flocode courses and the Beta Program. I am working hard on both. I will have more to share soon.
See you in the next one.
James🌊