Kafka Platform Engineer

Details:

• 12 Month Contract
• Remote (JHB or CT)

Position Overview:

Apply platform engineering expertise, critical thinking, design thinking and problem solving skills in an agile team environment to produce well-organised, optimised and documented source code that successfully delivers platform features & components of the service & solves complex technical problems with high quality solutions.

Minimum Requirements:

• Bachelors Degree in Information Technology 
• Hands-on experience with Kafka clusters hosted on AWS and on-prem
• Experience in building Kafka pipelines using Terraform, Ansible, Cloud formation templates, shells etc.
• Experience in implementing security & authorization (permission based) on Kafka cluster.
• Experience in setting standards to automate deployments using Kubernetes, Docker, Chef or Jenkins
• Experience in open source and confluence Kafka, zookeepers, Kafka connect, schema registry, KSQL, Rest proxy and Kafka Control center.
• Experience in Kafka Mirror Maker or Confluent Replicator
• Experience in High availability cluster setup, maintenance and ongoing support
• Hands on experience in standing up and administrating Kafka platform from scratch which includes creating a backup & mirroring of Kafka Cluster brokers, broker sizing, topic sizing, h/w sizing, performance monitoring, broker security, topic security, consumer/producer access management (ACL)
• Knowledge of Kafka API (development experience is a plus)
• Knowledge of best practices related to security, performance, and disaster recovery.
• Demonstrate a product mindset with an ability to set forward thinking and direction.
• Ability to synthesize large amounts of complex data into meaningful conclusions and present recommendations.

Responsibilities:

Platform Engineering, Deployment & Optimisation

• Leverage platform engineering expertise & a working knowledge of specific platforms and their constituent parts to positively contribute to platform feature and service designs 
• Take accountability for deeply understanding & building expertise on the platform, the platform contribution to the broader platform strategy & business and customer requirements & all available technologies 
• Develop business & customer aligned, lasting & innovative platform feature sets 
• Positively contribute to, define & implement the development/engineering lifecycle (end to end) and lead lifecycle improvements over successive releases. 
• Produce simple, extensible, and maintainable features with very few defects. Continually works to find and fix defects. 
• Leverage platform (service design & code) & engineering technical expertise to proactively identify risks and prevent defects 
• Identify patterns that reveal errors and unanticipated problems, and influence telemetry analytics design decisions 
• Identify & Apply metrics to drive the quality and stability of code 
• Ensure consistent, usable, forward-looking, maintainable test infrastructure 
• Apply object orientated design in development & draw from a large base of design patterns 
• Uphold our technical principles of: self-service, scalability, re-usability, resilience & stability in all development practices and deliver feature sets that sustainably deliver on these principles 
• Apply knowledge and expertise in User / Customer experience to deliver a stable user experience in all design & development that is simple, elegant, and useful. 
• Proactively identify development practices that work well & leverage & improve these for future solution design and deployment 
• Document feature set and constituent design information to produce solution design blueprints and validation collateral 
• Based on engineering & platform expertise and a sound knowledge of the business & customer requirements consistently provide key ideas for the enhancement & optimization of the platform 
• Develop & document complete designs and implement testing processes that identify and eradicate platform or service-wide problems / improve on platform performance
• Demonstrate a strong commitment to the quality & ongoing quality assurance of the product/service’s in use, and take appropriate action to resolve issues throughout the lifecycle of the product or service 
• Ensure alignment to platform and service release plans for effective deployment of solution designs 
• Drive & implement design review processes & practices with the team & participate & positively contribute to design reviews for the feature area led by others 
• Apply expertise in engineering & available technologies to enable effective review processes 
• Cascade review lessons learnt for the benefit of the broader team 
• Enable problem resolution across technical teams, apply and or facilitate root cause analysis, deeply understand the root causes of issues and find ways to resolve them (sustainably) 
• Continuously review & find ways to optimize team processes & improve engineering quality, productivity, and team responsiveness to feedback and changing priorities 
• Optimize, refactors and reuse components to improve performance and maintainability – ensuring maximum efficiency, effectiveness, and return on investment 
• Take accountability for maintaining platform standards and best practices, and drive adoption across multiple service teams 
• Demonstrate knowledge of, identify & find required data for the ongoing monitoring & improvement of the platform feature area 
• Proactively monitor the performance of the platform features & solutions 
• Translate monitoring data into clearly articulated descriptions of opportunities & defects, their impact to the customer scenario and to the product or service as a whole, and the relevance to product and service targets (within scope of the job). 
• Cascade and communicate these insights across the team to influence platform decision making 
• Leverage production performance monitoring and customer data to make feature area technical design and implementation decisions. 
• Synthesize data from multiple sources and present & define concise, impactful conclusions that influence feature teams to solve defects and meet quality targets

Financial & Cost Optimization, Risk & Governance

• Understand the platform cost model & the optimal cost : serve of the platform, use this understanding to drive the platform efficiency & effectiveness objectives
• Participate in incident management & DR activity – applying critical thinking, problem solving & technical expertise to get to the bottom of major incidents
• Apply Group risk (e.g. Architecture, development, change & release etc.), governance, compliance & regulatory standards and frameworks
• Proactively identify technical risks and mitigate these (pre, during & post release)
• Update / Design all feature documentation aligned to the organization technical standards and risk / governance frameworks

People

• Proactively & effectively (time & quality) collaborate & drive collaboration on features across feature team (e.g. managing dependencies & interdependencies, conflict resolution, information & best practice sharing, problem resolution outside of own area etc.)
• Apply collaboration skills in development practices & issue resolution regardless of team boundaries & lead communication with relevant stakeholders through to conclusion
• Contribute to a high performance team environment & culture as a high performing member of a self-directed team aligned to agile working
• Leverage coaching techniques in all team related activity to drive a higher quality service, design and deployment of technical solutions.
• Proactively seek & provide feedback across a range of stakeholders for the benefit of the team & leverage feedback to consistently improve platform solution delivery
• Coach & mentor other engineers & support engineering teams on technical solutions and problem resolution & proactively seek coaching & mentoring from others
• Participate in peer reviews, testing, problem solving within and across the broader team
• Cascade platform performance feedback and ensure capability development across teams to enable efficient & effective platform service delivery
• Drive the design of content, tools & materials for strategy alignment, education & adoption processes to be applied across stakeholder groups