I Tried Out the AWS Startup Prompts So You Don't Have To
Exploring AWS' new startup prompts for developers: A hands-on review of their effectiveness and usability.
In the ever-evolving landscape of cloud computing, AWS has introduced a new feature aimed at enhancing the developer experience: startup prompts. These prompts are designed to give developers a head start when creating new projects by providing pre-configured templates and suggestions. As a developer, I decided to put these startup prompts to the test to see how effective they are in real-world scenarios.
Rag Chatbot with Claude
For my first experiment, I chose to create a Retrieval-Augmented Generation (RAG) chatbot using AWS' startup prompts. The goal was to ,with little to no modification, use the prompt and see what we get.
One change I made was to switch from using Terraform for infrastructure as code to using AWS CDK, as I find CDK to be more intuitive and easier to work with.
To start I copied the prompt for the RAG chatbot with Claude. The prompt provided a comprehensive setup, including the necessary AWS services. It took twelve minutes for claude to set up the project.
The generated project included:
- A lambda directory with a basic lambda functions to handle chatbot requests.
- A CDK directory with infrastructure code to set up the necessary AWS resources.
- A frontend directory with a simple React application to interact with the chatbot.
- A deployment guide to help with the deployment process.
The lambda was written in Python while the CDK and frontend were in TypeScript. The project structure was well-organized, making it easy to navigate and understand.
Following the Deployment Guide
The deployment guide was straightforward and easy to follow. It provided clear instructions on how to set up the project, including installing dependencies, configuring AWS credentials, and deploying the infrastructure using CDK.
During the deployment process, I encountered a few minor issues. The first involved creating Lambda layers for dependencies. While the instructions described this as a production optimization, it was actually required for deployment. Unfortunately, the prompt provided no guidance on how to create these layers, so I had to research the process independently to move forward. Since I don’t typically work with Python and wasn’t targeting a production environment, I chose to bypass this step by commenting out the dependency layer in the CDK code.
The next issue was related to how the CDK code referenced Lambda assets. It used a relative path assuming the functions were located within the CDK directory, but the project was structured as a monorepo, with the Lambda functions placed at the root of the project. This mismatch caused deployment issues that required path adjustments.
Next, I encountered a problem with gitignore files. The output included a .gitignore file in the root directory, but it included a line to ignore /lib as a Python rule but that conflicts with the AWS CDK output. To fix this, I changed the directory from cdk/lib to cdk/stacks to keep the Python rule while avoiding conflicts with CDK.
Finally, the prompt hallucinated a lambda function named chat-handler that it defined in the infrastructure code but did not actually create it in the lambda directory. To fix this I provided the following prompt: @cdk/stacks defines a lambda function named chat-handler but there is no such file @lambda. Unfortunately, this caused Claude to check and re-write every file in the lambda directory, which was not ideal.
After this I was able to successfully deploy the project using CDK. Not so fast - I then encountered an issue during deployment where the CDK stack failed to create our rag-chatbot-network-policy due to AllowFromPublic being false when it must be a constant value true.
Finally, it deployed. This process took about 2 hours in total. One big loss of time was due to setting the deletion policy of the DynamoDB table to RETAIN to avoid data loss during testing. This caused the stack to fail to redeploy because the table already existed, leading to additional troubleshooting and manual deletion of the table.
Configuring and Redeploying
After the initial deployment, I needed to configure the frontend application to connect to the deployed backend services. The deployment guide provided instructions on how to use the AWS CDK outputs and integrate them into the React application. This lead to another issue - the deployment guide wanted us to manually deploy the frontend using the aws cli, but the AWS CDK code is capable of deploying the frontend as well. Interestingly enough, Claude imported the s3 deployment package but did not use it. I modified the CDK code to include the frontend deployment, and redeployed.
Redeploying the stack required that I manually run the build command in the frontend directory to generate the production-ready files. This failed due to a type error in the React code. Claude tried to configure AWS Amplify for authentication, but did not set it up correctly. To fix this, I had to modify the React code to fix the type error.
This also highligted another issue - Claude did not set up the build script in the frontend to reference the environment variables so I added a new build:local script to the package.json file that sets the environment variables before running the build command.
Auth Flow and the First Major Bug
With the frontend deployed, I accessed the React application in my web browser. The initial load was quick, and the user interface was clean and intuitive. However, when I attempted to register in using the authentication flow, I encountered a significant bug. The login process failed, and I received an error message indicating an issue with my JWT token. Upon further investigation I realized that the authentication flow was set to USER_SRP_AUTH but the frontend was expecting a token. To fix this, I prompted Claude with @frontend/src/App.tsx expects user?.signInUserSession?.idToken?.jwtToken but the cognito user is missing that field. review @cdk/stacks/rag-chatbot-stack.ts to see if cognito is configured properly.
Claude then made some modifications but they were incorrect. I needed to tell Claude to review recent documentation for Amplify to properly request a token. After this change, I was able to successfully register and log in to the application but I could not connect to the websocket API. This was due to the way the Python Lambda function was packaging dependencies. The error is Runtime.ImportModuleError. After telling Claude about this error, it modified the CDK code to properly build the Lambda and redeployed the stack. This would become a recurring theme - I would tell Claude about errors in the build process, and it would modify every lambda function to fix the issue. I had to explicitly tell Claude to only modify the function that was failing.
The First Document Upload
With the authentication flow working and the websocket connection established, I was able to upload documents to the chatbot. It was at this I met my next major hurdle. The document upload feature uses a presigned URL to upload files directly to an S3 bucket. However, when I attempted to upload a document, I received an AccessDenied error. After investigating the issue, I realized that the IAM role associated with the Lambda function did not have the necessary permissions to generate presigned URLs for the S3 bucket.
Now that the document upload feature was working, I tested the chatbot's ability to retrieve and respond to queries based on the uploaded documents. I uploaded a few sample documents and then asked the chatbot questions related to the content of those documents. The chatbot failed to process the documents correctly. After investigating the issue, I realized that the Lambda function responsible for processing the documents was not correctly configured with the data access policy.
Configuring Model Access
After fixing the processing permissions issue, I encountered another problem related to the model access policy. I needed to log in to the AWS Management Console and navigate to the Bedrock service to request access to the required models. The access request was answered almost immediately, and with that done, I uploaded a new document only to find that document processing lambda still failed. After investigating the issue, I realized that the Lambda function was calling the bedrock client incorrectly. To fix this, I prompted Claude to use the serverless invocation pattern for the bedrock client. After this change, I was able to successfully process documents and retrieve relevant information using the chatbot.
With the document processing and retrieval features working, I tested the chatbot's ability to answer questions based on the uploaded documents. I asked the chatbot a series of questions related to the content of the documents, and it was able to provide accurate and relevant responses.
Missed Opportunities
One area where the prompt could be improved is for it to provide instructions about using latest libraries and best practices. For example, the prompt used an outdated version of AWS Amplify in the frontend code. Using the latest version would provide better performance and more features. Additionally, the prompt included instructions on how to set up monitoring and logging for the deployed resources, when it could have included those monitoring and logging configurations directly in the CDK code. This would have saved time and effort during the deployment process but can easily be fixed by prompting Claude to add monitoring and logging.
Conclusion
Overall, my experience with AWS' startup prompts for creating a RAG chatbot with Claude was a mixed bag. While the initial setup and project structure were well-organized and easy to navigate, I encountered several issues during the deployment and configuration process that required significant troubleshooting and manual intervention. However, with some persistence and problem-solving, I was able to overcome these challenges and successfully deploy a functional RAG chatbot. The final code is available on GitHub and for a limited time (a day or two), you can access the deployed application.
Reflecting on the experience, I believe that while AWS' startup prompts provide a solid foundation for building cloud-based applications, there is still room for improvement in terms of documentation and guidance. More detailed instructions and troubleshooting tips would be beneficial for developers who may not be as familiar with certain AWS services or deployment processes.
Furthermore, it's important to note that while the startup prompts can help accelerate the initial setup process, they are not a substitute for a deep understanding of the underlying technologies and services being used. Developers should still take the time to familiarize themselves with the AWS services involved in their projects to ensure they can effectively troubleshoot and optimize their applications.