As a former founder, President, and later a CEO of software businesses (and current consultant), I’ve always valued direct access to data for independent analysis. I prefer to avoid delays associated with BI or data science teams, as I find that hands-on iteration with data leads to deeper business insights. I learned SQL and Python (especially Pandas) in order to quickly extract insights. However, even with this expertise, the complexity, inconsistencies, and sheer volume of data within a new data warehouse can still present significant challenges. That was the case with my latest consulting project.

The project presented an additional layer of complexity: I needed to apply advanced techniques, specifically graph neighborhood detection algorithms, to unravel intricate relationships within the data. This was a new frontier for me, having a general machine learning background but no prior experience with graph analysis.

This is where Large Language Models (LLMs) proved to be invaluable. I found them to be powerful allies in overcoming these dual challenges – understanding the data landscape and mastering novel analytical techniques.

Cracking the Client’s Data Code with Claude Code

For navigating the client’s data warehouse, Claude Code became an indispensable assistant. It significantly accelerated my data understanding process. I maintained a “rolling data schema” document, a live record where I’d continuously append new findings about the schema and underlying data as I uncovered them. Claude Code helped me quickly generate exploratory SQL queries, allowing me to formulate a series of precise questions for the data engineering team. When I eventually met with them, I arrived armed with a comprehensive schema document and sharp inquiries, enabling us to swiftly pinpoint the critical data required for my analysis.

Unlocking Graph Algorithms with ChatGPT

Separately, ChatGPT was instrumental in helping me grasp the novel graph analysis techniques. It quickly guided me to the most relevant class of algorithms for my specific use case and aided in choosing between them. It also facilitated an intuitive understanding of these complex techniques and even assisted with hyperparameter tuning, allowing me to build the most robust analysis possible within the given timeframe.

LLMs: Wise Fools that Need Structured Guidance.

Without the assistance of these LLMs, I wouldn’t have been able to complete this analysis within the project’s demanding timeline. They acted as a significant force multiplier, allowing me to bridge knowledge gaps and accelerate my workflow.

However, it’s crucial to understand that LLMs are not a panacea. They are, as I’ve come to see them, the smartest “dumb” people you’ll ever meet. They can generate highly plausible but ultimately incorrect assumptions if not carefully managed. To keep them on track, I developed a few key strategies:

Ground Them in Data: Provide LLMs with constant access to your rolling schema document. Feed them “SELECT *” outputs from major tables so they can inspect the actual data. Maintain a dynamic document outlining your notebook and its structure, including: All created dataframes, associated columns for each dataframe and few sample rows of data, if feasible.

Establish Rules of Engagement: Treat LLMs as knowledgeable collaborators, not autonomous agents. The following approaches made my most recent project go more smoothly.

• No Unilateral Changes: Do not allow LLMs to make changes independently.
• Pre-Code Discussion: Always discuss changes and your approach with the LLM before they write any code.
• Read-Backs: If you propose a change to the approach that the LLM suggest, have them “read back” what you just told them to ensure comprehension.
• Drill Down: If something doesn’t make sense, drill into the specifics with follow-up questions.
• Rigorous Code Review: Once code is generated, thoroughly review it and question every line – LLMs seems to get tripped up on merge logic and filtering more than I would expect.
• Managed by Outputs: Over three decades ago, as a principal in a management consulting firm, I mastered the art of managing by outputs. This approach involves scrutinizing numerical data presented by an engagement team, specifically looking for anomalies. Often, a “strange” number indicates a methodological flaw that requires immediate attention. For instance, does the total revenue figure align with the canonical finance report? If not, a discrepancy likely exists. Similarly, if the relative churn rates across different segments appear illogical or the overall churn level seems off, further investigation is warranted. Effective management by outputs necessitates a strong intuitive grasp of the data. This intuition is developed by reviewing numerous company reports prior to conducting analysis. Such a proactive approach not only sets client expectations but also ensures that findings are directly verifiable and consistent with established financial records.

LLMs proved to be an excellent tool, significantly contributing to achieving a favorable outcome within a tight deadline. Nevertheless, it’s crucial to establish control over the process before commencing analysis; neglecting this step will likely lead to difficulties, making for a rough ride.