Performance Test Engineer USC

Performance Test Engineer

Duration: Full Time Hire

Location: REMOTE

Type: W2

Program Description:

This program provides IT services focused on building, securing, and operating the Department of Veteran Affairs LGY's home loan product-line technology. The contract's purpose is to modernize and sustain critical home loan technology systems that support LGY's delivery of mortgage-related services to program stakeholders, to provide continuous delivery and security integration.

Position Description: The Performance Developer will support the Department of Veterans Affairs (VA) Loan Guaranty Service (LGY). The Performance Developer will be responsible for building the observability & monitoring artifacts, telemetry and logging that supports serverless system performance, diagnosing performance bottlenecks, and supporting performance optimization efforts across the LGY platform. The Performance Developer will not just write Lambda code they will own the engineering pattern for how LGY Lambda services are built, instrumented with AWS ADOT, deployed through CI/CD, tested, and monitored in Dynatrace & Splunk. This role works closely with development first and then assists operations teams to ensure system performance, scalability, and reliability across Salesforce and AWS-based integration services. The ideal candidate must have development experience working with API-driven systems, cloud infrastructure, and performance monitoring tools to develop & analyze telemetry, logs, and system metrics to identify and resolve performance issues.

Years of Experience: 4+ years

Education Requirements: Bachelor's Degree

Responsibilities:

• Design, build, & maintain AWS Lambda services in Node.js & Python based serverless portfolio.
• Lead AWS ADOT-based instrumentation for Lambda workloads, including OpenTelemetry layer adoption, configuration, OTLP settings, & runtime compatibility validation.
• Using GitHub Actions, develop pipelines to standardize CDK Lambda constructs for Dynatrace & AWS ADOT libraries, reusable layers, environment variables, active tracing, Lambda Insights, and centralized log configuration.
• Must have dev background experience implementing & supporting end-to-end tracing, metrics development and structured logging across API Gateway, Step Functions, SQS, S3, RDS/Postgres, Salesforce integrations, Oracle/SOAP services, document APIs.
• Validate Dynatrace instrumentation compatibility for Node.js Functions & implement wrapper or alternate instrumentation strategies where required.
• Develop and maintain correlation ID and trace propagation across Lambda, Step Functions, and downstream service calls so transactions are observable end to end.
• Enforce observability practices for AWS cloud workloads, including PII-safe structured logging.
• Build & maintain GitHub Actions pipelines for linting, unit tests, coverage, deploy, and post-deployment telemetry validations.
• Create automated tests that verify instrumentation is working, including trace/log/metric arrival in Dynatrace & Splunk, cold-start visibility, and alert validation in each environment.
• Develop Python-based solutions to capture Lambda errors, duration, throttles, retries, cold starts, workflow failures, and downstream latency.
• Tune Lambda memory, timeout, concurrency, packaging, and network configuration to balance AWS ADOT overhead, Dynatrace visibility, reliability, and cost.
• Modernize older Lambda implementations and pipelines to a common AWS ADOT plus Dynatrace standard across multiple environments.
• Investigate performance incidents and collaborate with development teams to diagnose root causes.
• Review system logs, API metrics, and AWS service telemetry to identify infrastructure or application performance constraints.
• Provide performance optimization recommendations to development teams when performance bottlenecks are identified.

Required Skills:

• Bachelor's degree in Computer Science, Information Technology, Engineering, or related field or equivalent professional experience.
• Development background experience with Node.js and Python on serverless architecture. THIS IS A MUST.
• Years of experience in performance engineering, application performance monitoring, or system performance analysis.
• Experience with Dynatrace and Splunk.
• Experience analyzing application performance metrics and system telemetry.
• Must be able to decipher code for troubleshooting performance issues in API-driven or distributed systems.
• Familiarity with cloud-based environments such as AWS.
• Strong analytical and problem-solving skills.
• Experience working with monitoring tools and log analysis platforms.

Preferred Skills:

• Experience supporting VA or federal government IT systems.
• Experience with AWS services such as Lambda, API Gateway, and cloud monitoring tools.
• Familiarity with Salesforce-based applications and enterprise API integrations.
• Experience supporting systems that process sensitive or regulated data.