New - Software Engineer, Data Platform - En - Mep

Job Description Building trust for a seamless society. We challenge ourselves daily to achieve this mission. XXX business started as a payment service, but that was just our first step towards achieving the mission we envision. We want to create a society where anyone can realize their dreams through a new ecosystem centered not only on this payment service, but on credit.

Let's work together to update society's deinition of "payment," "credit," and society itself. Our Culture XXX culture has been central to the company since our founding, with the three core values of Go Bold, All for One, and Be a Pro. In addition, the “Trust & Openness” value was introduced to embody our culture of mutual trust, where information remains transparent and rules are kept to a minimum.

Regardless of how big XXX gets, the culture will remain essential to achieving our mission and is evidenced throughout our organization. Work Responsibilities XXX Group utilizes data for business decisions and other processes in a wide variety of areas, including marketing, machine learning, and R&D. As part of XXX Data Platform Team, you will design, develop and operate the data infrastructure/data pipelines which support data utilization for the entire XXX Group, including not only XXX, but the XXX marketplace app.

You will also be responsible for work on the XXX Group’s data utilization as a whole. The Data Platform Team is an engineering organization which builds the systems necessary to a wide range of domains that utilize data. This work includes development of an ecosystem enabling data collection, Data Lake, DWH, use of collected data and other processes required by data utilization.

The following describes the responsibilities of a software engineer on the Data Platform Team, as well as the environment and organization in which they work: Job Description Collecting data from microservices Developing batch-based data pipelines Developing streaming-based data pipelines Developing SDK for logging within microservices Data Lake, DWH Constructing Data Lakes for storing collected data Managing partial permissions and resources on DWH Data utilization Developing tools to support use of data in Data Lakes and on DWH Developing a platform to provide stream data processing function Company-wide Improving performance, aiming for low latency/high throughput according to application and middleware development, operation, and requirements Developing tools to automate operations and/or lower costs Identifying and solving technical issues with the system caused by engineering Tools Used/Dev Environment Programming languages: Python, Java, Scala Infrastructure: Google Cloud Platform (Compute Engine, Kubernetes Engine, Dataflow, Pub/Sub, Cloud Storage, Big.

Query, Spanner, Cloud SQL, Cloud Logging etc.) Distributed processing: Apache Flink / Apache Spark Workflow: Apache Airflow BI tools: Looker / Data Studio Configuration management tools: Terraform Monitoring: Stackdriver Monitoring / Datadog / Pager. Duty CI/CD: Circle CI / Spinnaker Knowledge Tool: Crowi / Confluence Other: GitHub / Slack Development Method: Agile (Scrum, Kanban, etc.)

Dev Organizations As part of the Data & Machine Learning Team, you will work on development alongside members with a broad range of expertise from the following organizations. Teams Data & Machine Learning Platform Data Management Machine Learning (Credit) Machine Learning (Anti-Fraud) Bold Challenges XXX to build large-scale data infrastructure to support the massive amounts of data generated by XXX Group, including the XXX Marketplace app with its over 18M monthly users.

Work on development of an ecosystem to support data-driven business expansion by addressing the data utilization needs of many domains including analytics/decision-making, marketing, machine learning, and R&D. Take on the entire data platform creation process, from design to development to operation, to revise existing data infrastructure and develop new features which meet new data utilization needs.

Required Experience A shared belief in XXX and XXX missions and values A degree in computer science or a related field, or else five or more years of practical experience in software development Experience designing, developing, and operating large-scale services and/or distributed systems At least two of the following: Experience developing in the cloud with AWS, GCP, etc.

Experience developing systems utilizing container technologies such as Kubernetes Development experience using message queues like Cloud Pub/Sub and Apache Kafka Experience in data processing development using distributed processing frameworks like Apache Flink and Apache Spark Experience with ETL system using workflow engines like Airflow and Digdag Experience designing application logs for large-scale services Preferred Experience Experience working at a financial institution, Fintech company, or EC company Ability to identify the cause of technical system issues (e.

g. drop in performance) and to resolve them Knowledge of network protocols such as TCP/IP, HTTP, gRPC, etc. Experience developing and operating software using two or more of the following: Go, Java, Scala, or Python Developed data collection systems using Treasure Data-created OSS (such as Fluentd, Embulk) Experience in development using DWH like Big.

Query, Redshift, or Snowflake Experience in development using RDBMS, including MySQL and RDS Experience in development or cluster operation using the Hadoop ecosystem Experience developing software based on a microservice architecture Experience publishing and contributing to OSS Screening Criteria Has a strong, shared belief in XXX and XXX missions and values Capable of going bold and possessing a strong sense of responsibility in regards to their work Possesses excellent communications skills which can push organizations and projects forward Able to collect information independently and identify the essence of issues to solve problems through engineering Thinks independently, acts independently, and leads operations with All for One mentality for the sake of the team and while holding on to professional mindset Technical Assessment Criteria Documentation skills Testing Ability to code (efficiency, readability, maintainability) Design aptitude Language Requirements English: Business level (preferred) Japanese: Business level