#031 - AI Tools for Engineers - Part 1
Evaluating Large Language Models in Civil and Structural Engineering
Introduction
We’re all looking for ways to improve our workflow, increase efficiency, and reduce costs; this is inherent in engineering. However, over the last 18 months, we have been power-hosed by AI marketing and hype. While it would be foolish to deny AI's potential, we need to approach this topic with a healthy dose of pragmatism.
In this article, I'll share my personal experience using AI and large language models (LLMs) for civil and structural engineering tasks and explore their benefits and limitations.
The Adoption of AI in Construction and Engineering
There was a recent corporate event in Florida (2024 AI Summit by BST Global), where AEC industry leaders gathered to discuss the role of AI in the construction sector. There were some interesting insights to take away.
In my opinion, the general tone of the conference was quite executive-focused, with limited technical information or real-world substance.
High-Level Overview:
The event provided a high-level overview of AI and its benefits.
Many companies are still figuring out how to apply AI in meaningful ways but are reluctant to be the canary in the coal mine.
Data Control in AEC Companies:
Big tech companies are aggressively trying to access data from AEC companies, but there's widespread resistance.
AI Investment:
The consensus among the talks was clear: investing in AI is crucial.
Speakers were tight-lipped about specific applications to protect strategic advantages.
Case Studies:
Some impressive applications of AI were showcased, such as converting hand-drawn sketches into 3D models.
These examples were largely conceptual and lacked the precision required in traditional engineering design but were suitable for architecture, form-finding or ideation.
Resource Allocation:
Many companies are actively dedicating significant resources to developing AI capabilities, with some larger companies establishing dedicated AI task forces.
Significant portions of IT budgets are being allocated to AI development, with suggested increases of at least 30% being mentioned.
Despite the optimism, the industry is clearly still in the early stages of exploring AI's potential. We all recognize its importance, but things are still taking shape.
Personal Experience with AI in Civil and Structural Engineering
I’ve been playing with AI models since they were publicly available, and at this point, they've become an indispensable tool in my workflow. I've experimented with several of them, including local LLMs, and have been impressed by their capabilities. For instance, I've used them to help with specification writing, document preparation, design calculations/automation and code development. The benefits have been tangible, and I believe that AI has the potential to revolutionize the way we work as engineers.
It should go without saying that we must take full responsibility for any engineering output produced by AI. Relying on it for accuracy is insane. We are still on the hook until the legal frameworks are developed to ensure that OpenAI or Google will take ownership over errors and omissions. I don’t see this changing any time soon.
Like a chainsaw, use with caution.
Overview of Local LLM Models
One of the most exciting developments in AI research is the emergence of local LLMs, such as llama, Mixtral, Gemma and most recently Llama3. These models are remarkable because they can be run locally on your machine without relying on cloud services, subscriptions or expensive hardware.
What's more, they have become surprisingly fast and effective. With local LLMs, you can process vast amounts of data, generate text, and even perform tasks normally requiring human-level intelligence. Of course, it's essential to understand the limitations of these models, but with the right approach, they can be a powerful tool in your workflow.
Check out Groq to test the latest Llama3 model (70b) at blisteringly fast speeds without installing anything. Note if you install locally, you’ll want the much smaller (8b) model, or it will run at a snail’s pace.
For those interested, there are some fantastic open-source projects focused on managing local and remote LLMs.
Applications of Local LLM Models in Civil and Structural Engineering
One of the most significant benefits of local LLM models is their versatility. They can be applied to a wide range of tasks in civil and structural engineering, from routine administrative tasks to complex design and analysis. You can also enjoy the important security aspect of maintaining control of your data.
I've used local LLM models to:
Assist with specification writing, generating boilerplate text and helping to ensure compliance with industry standards, using RAG models trained on regional specifications to compare and review accuracy.
Automate document preparation, including report generation and triggering workflow automation with Activepieces.
Streamline calculations, writing/testing and debugging in Python. Claude 3 and, more recently, GPT-4o are the best for this.
Helped with administrative tasks, such as email management, note/meeting transcription, calendar organization, and automating specific workflows and documentation processes.
The possibilities are far-reaching. These models can change how we work as engineers, especially if you have a specific workflow or systematic process that you understand fully from beginning to end. This may seem obvious, but sometimes it’s not.
Fine-Tuning Local LLM Models for Engineering Tasks
One of the most exciting aspects of local LLM models is their ability to be fine-tuned for specific tasks. By providing the model with custom instructions and examples, you can tailor it to your specific needs and workflows. For instance, you could fine-tune a local LLM model to:
Generate custom code snippets for common engineering tasks or based on a corpus of reference data for a specific Python library or engineering software that you use.
Assist with data analysis, checking and visualization.
Even help with creative tasks, such as generating ideas for innovative designs.
Clean up voice transcriptions/meetings into legible and actionable notes.
The key is understanding how to work with these models, developing good prompting skills, and recognizing their limitations.
With practice and patience, you can do a lot.
Structural Work using Llama3
I have been actively developing two AI models built on Llama. These models are designed to assist with specific tasks in structural engineering, such as generating code for finite element analysis and automating the design of structural components.
One is trained on the SAP2000 OAPI documentation with many examples, including the SAP documentation. It’s been helpful as an assistant to complete specific tasks and help debug code for FE analyses.
Another model uses a system prompt to create a panel of ‘agents’ who cross-check and validate each other's responses. This one is interesting because this kind of self-appraisal loop seems to result in higher-quality responses.
One key challenge in developing these models has been ensuring that they are accurate, reliable, and efficient. This requires fine-tuning the models and extensive testing and validation. This is a moving target as the leaps in ‘intelligence’ between model releases have been staggering.
The results so far are promising, and I believe that these models have the potential to significantly streamline my processes and workflows.
I have some stuff related to this on my GitHub and on Ollama, for anybody who’s curious, but it’s a work in progress.
Issues and Bottlenecks in AI Model Development
Despite the incredible potential of AI models in engineering, there are still significant challenges and bottlenecks that need to be addressed. One of the biggest issues is the lack of high-quality, domain-specific training data.
Many AI models are trained on huge general-purpose datasets, which may not be relevant or accurate for engineering applications.
Another challenge is the computational resources required to train and run these models. While local LLM models like Llama (8b) are more efficient than cloud-based models, they still require significant processing power and memory. This can be a barrier for smaller engineering firms or individual engineers who don’t have access to high-performance computing resources.
One of my training experiments went off the rails when I tried to create a Retrieval-Augmented Generation (RAG) model based on ACI 318-19 (US Concrete Design Code). Soon after OpenAI released their API and Playground, I attempted to scan a PDF document to generate embeddings for consulting the ACI concrete code. However, the process became expensive quickly. I used Optical Character Recognition (OCR) libraries, which are necessary for recognizing Greek symbols and other complex notations. Due to the document’s graphs, tables, and images, the OCR process was more complicated and resource-intensive than anticipated. I forgot to set a spending limit on my API, resulting in a big chunk of cash spent in an hour. It could have been a lot worse.
Some larger companies likely have impressive proprietary datasets to make practical machine learning or RAG models. Whether or not this is worthwhile is hard to say at this point. Even if a company has great proprietary data on a specific niche—say, concrete performance in extremely hot climates—I think it's only a matter of time before higher-fidelity datasets become more widely available. So, how much effort should they put into cleaning and refining their data and tuning their models? Is it worth it?
Companies need to decide how to approach their proprietary information. Maintaining a commercial edge is inherent in all industries and separates the high performers from others. However, as the flow of data increases its velocity, this type of data hoarding will become more difficult and less impactful.
This is a broader, nuanced discussion with many different ingredients that are beyond the scope of this article, but it's food for thought nonetheless!
AI Models - My Thoughts and Experiences
After experimenting with most of the commercial AI options as they've been released, I've noticed a few different things. I have summarized my thoughts and observations below, but please keep in mind they are subject to change as I learn more or as models are updated. This is just a snapshot of my current opinion.
Usage: General-purpose, this is my all-round workhouse for most tasks.
Pros:
Highly flexible and versatile
Improved speed with recent updates (GPT-4o)
Multimodal capabilities (documents, text, voice)
Paid version provides data protection (allegedly)
Cons:
Inconsistent performance, at times, can be slow and produce useless outputs.
Text output can be very generic AI style, requiring stringent prompt engineering.
Key Observations:
GPT-4o is the go-to tool for most tasks. A recent update has improved speed and flexibility. Text quality often requires careful prompt crafting.
Usage: Writing, logical reasoning, coding assistance
Pros:
High-quality text output
Succinct and pragmatic responses
Impressive logical reasoning
Large input text window
Good for code debugging and writing
Cons:
Slow at times
Limits on use cases
Premium version is not available in Canada, I need to use the API
Key Observations:
Excellent for writing and logical tasks. Helpful for code refinement. Accessible via Claude Opus API in Canada.
Usage: Research and information retrieval
Pros:
Quick, natural language responses
Aggregates and summarizes information
Provides hyperlinks to references
Cons:
Free version limits
Key Observations:
Great for research and finding specs. Functions like an AI-powered search engine. Free version works well for most needs.
Usage: General-purpose (initially promising)
Pros:
Initially, better writing than GPT-4
The interface looks pretty nice
Cons:
Simplistic responses with excessive explanations
Clunky interface
Annoying inefficiencies
Key Observations:
Initially impressive but became less useful over time. Simplistic responses and clunky interface reduce usability.
Usage: Free model, experimentation
Pros:
Impressive performance
Open-source project (Open Web UI)
Can run multiple AI models locally
Grok API for large parameter models. Be cautious with data security on Grok.
Cons:
Limited by hardware capabilities
Potential data security concerns when using Grok
Key Observations:
Great free tool for experimentation. Openwebui provides a comprehensive chat interface. This is a fantastic tool that runs in a docker engine and allows you to have all of your AI models in one place.
The Future of AI in Engineering
As AI evolves, we will see even more powerful and sophisticated models emerge in the engineering domain. These models will likely handle more complex tasks, like design optimization, simulation, and even autonomous decision-making.
Many questions about security, regulation and data protection are yet to be answered. We are currently in a wild west of AI innovation. It’s exciting but the latent conspiracy theorist in me can’t help but ponder the ‘what ifs’!
AI is not a silver bullet; we’re a long way from replacing the need for human expertise, judgment and liability. Instead, I see AI as a powerful tool that can augment and enhance the work of engineers, allowing us to work more efficiently and effectively.
There are many interesting philosophical discussions around AI and its implications for our profession, which has traditionally emphasized civic duty and public safety. I’m intrigued to see how regulators integrate these tools into our legal and ethical responsibilities. This landscape is still evolving, and we need to watch it carefully.
While a T1000 is not likely to replace you in your current engineering role, the next wave of engineers and graduates familiar with AI will quickly develop skills that make them incredibly efficient and productive. They will be able to provide equivalent value at a much lower cost.
So, don't sleep on this. AI is here, and you need to understand how it fits into your industry, niche, and role. It's another skill, just like Googling.
Conclusion
I’m curious to hear what other engineers think. In my experience, I’ve encountered a broad mixture of responses and attitudes towards AI: lots of hope, some fear, and some indifference. Where do you stand?
The Beta Program is ready to roll but I am also expecting a baby, well technically I am expecting my wife to have a baby so I plan to wait until the dust settles but applications are open for those eager to get on the list.
I will share a dedicated post soon that will explain the Beta Program and its intent in more detail.
Thanks for your attention, I appreciate it.
Until then, keep innovating.
James 🌊
Please share some activepieces workflows?