Request middleware performs some specific function on the HTTP request or response at a specific stage in the HTTP pipeline before the business logic / app handler.

Currently middleware is a common term but in different programming language or framework sometimes it called filter. For illustration could be like this

As you can see, middleware could be layered for more than one before hitting app handler or business logic then later it will return the response. HTTP Request comes in to first layer or middleware, check the necessary and if all conditions are passed then it allowed to continue the journey. If failed, it could also returned the failed response anyway.

Basic Middleware

Let say we have simple HTTP API server

func rootHandler() http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        resp := defaultResponse{
            Status:      "OK",
            Description: "Success",
        }

        j, _ := json.Marshal(resp)
        w.Header().Add("Content-Type", "application/json")
        w.Write(j)
    })
}

func main() {
    mux := http.NewServeMux()

    mux.Handle("GET /", rootHandler())

    s := &http.Server{
        Addr:    ":8080",
        Handler: mux,
    }

    // ...
}

if we run the program go run main.go and visit http://localhost:8080 it will return 200 success with JSON response:

{
  "status": "OK",
  "description": "Success"
}

Now lets create our first middleware. Our middleware is to write a simple log that indicating the request is passing through. The middleware should be a function like this

func logMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        log.Print("Executing logMiddleware")
        next.ServeHTTP(w, r)
    })
}

Then we will modify the mux handler to use the middleware

func main() {
    mux := http.NewServeMux()
    mux.Handle("/", logMiddleware(rootHandler()))
    // ...
}

The result is when we’re visiting the API, it also wrote the log that already specified in the middleware logic.

Adding More Middleware

Interesting. Now try to add more middleware. This time we will check if the API request header contains x-signature, if not exists then we will return error.

Lets create the middleware function

func requestSignatureMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        log.Print("Executing requestSignatureMiddleware")
        h := r.Header.Get("x-signature")
        if h == "" {
            resp := errorResponse{
                Status:  http.StatusUnauthorized,
                Error:   "INVALID_MISSING_SIGNATURE",
                Message: "missing request signature",
            }

            j, _ := json.Marshal(resp)
            w.Header().Add("Content-Type", "application/json")
            w.WriteHeader(http.StatusUnauthorized)
            w.Write(j)
            return
        }
        next.ServeHTTP(w, r)
    })
}

We want to call logMiddleware first then continue to requestSignatureMiddleware. Our mux handler would be look like this now

func main() {
    mux := http.NewServeMux()
    mux.Handle("/", logMiddleware(requestSignatureMiddleware(rootHandler())))
    // ...
}

Here’s the result of the combined middlewares

• First request without x-signature header will returned 401 HTTP status

• Second request with additional x-signature header passed middleware logic and returned 200 HTTP status.

Cleaning Up

You may notice that our mux handler with multiple middlewares is little bit messy because it is calling function into function. What if we have many middlewares?

logMiddleware(requestSignatureMiddleware(authMiddleware(rateLimitMiddleware(rootHandler)))))

Hard to see.

Here’s what we can do, create a chain function to loop all middleware functions. You can named the function whatever you want.

// Middleware type for cleaner middleware chaining
type Middleware func(http.Handler) http.Handler

// Chain creates a single middleware from multiple middlewares
func Chain(middlewares ...Middleware) Middleware {
    return func(next http.Handler) http.Handler {
        for i := len(middlewares) - 1; i >= 0; i-- {
            next = middlewares[i](next)
        }
        return next
    }
}

Initiate the Chain with our middlewares and finally call it in our mux handler.

func main() {
    mux := http.NewServeMux()

    // Create a chain of middlewares
    middlewareChain := Chain(
        func(next http.Handler) http.Handler { return logMiddleware(next) },
        func(next http.Handler) http.Handler { return requestSignatureMiddleware(next) },
    // Add more middlewares here as needed
    )

    mux.Handle("GET /", middlewareChain(rootHandler()))
    // ...
}

Now it’s more manageable and it will still got same result as before.

So here’s the full code of this experiment https://gist.github.com/didikz/98de868f20887285f5aabc80f4dddcfa

Happy coding!

Graceful shutdown is safer and proper way to shutdown your HTTP server especially in Go where we usually managed the server script by ourself.

The “Graceful shutdown function” facilitates this process, allowing the server to transition smoothly without abruptly terminating active connections.

The most basic and simple way to run the HTTP server in Go is like this

s := &http.Server{
    Addr:           ":8080",
    Handler:        mux,
    ReadTimeout:    10 * time.Second,
    WriteTimeout:   10 * time.Second,
}

log.Fatal(s.ListenAndServe())

s.ListenAndServe() is called to run the server and listen incoming http requests. Yes this works, but when we’re talking about production system, we need more proper way using graceful shutdown to prevent unexpected behavior when the server needs to be stopped.

Here’s how graceful shutdown would be

    s := &http.Server{
        Addr:    ":8080",
        Handler: mux,
    }

    stop := make(chan os.Signal, 1)
    signal.Notify(stop, os.Interrupt, syscall.SIGTERM)

    go func() {
        if err := s.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
            log.Fatal(err)
        }
        log.Println("Stopped serving new connections")
    }()

    <-stop
    log.Println("Shutting down gracefully...")

    shutdownCtx, shutdownRelease := context.WithTimeout(context.Background(), 10*time.Second)
    defer shutdownRelease()

    if err := s.Shutdown(shutdownCtx); err != nil {
        log.Printf("server shutdown error: %v\n", err)
    }

    log.Println("Server stoppped")

It’s using goroutine and channel to notify server if there is termination signal. More example of this implementation at my fun project simple real-time system monitoring using Go & HTMX.

Happy coding!

Actually another offer came within the same week, made me took quite hard to choose which offer that will be best for me. By this decision I withdrawn the other offer.

Offering Considerations

I got 3 offers from companies and let me breakdown some points of them

D Company

• One of my ex employers so I know most of their working cultures

• The position is Head of Engineering

• The salary would be significantly dropped from my current salary

• On site but only around 10 minutes from my home

• Has some interesting management problems that could shape my managerial skills more

S Company

• Singapore based company in pharmacy business

• Latest funding was Series B and quite stable

• The role is senior associate full stack, would be PHP & React

• Base salary almost same with my current salary

• Work device provided by company

• Additional private insurance as work benefit

• Full remote

O Company

• Singapore based company, social commerce SaaS

• Relatively new founded company. Latest funding was pre-seed but they said current business is good and quite sustainable

• Will expand the business to Indonesia and I am their first hire as Indonesian Developer

• The role is senior PHP engineer

• Base salary is higher than current salary

• No work device (BYOD)

• Full remote

As you can see the last two offers are close enough with my career goal as I still want to pursue IC track rather than managerial track although in the future I probably want to change my direction to managerial role, but that’s not my current priority.

At first, I tent to S company as my next company if I passed the hiring process because the stability and I impressed by their leader. Then the O company gave me an offer with higher salary also they showed me many extra effort just to get me hired.

Spent my whole weekend to think about it because for me it was hard to decide. Then I decided to join the O company and withdrawn the offer from S company because:

• more higher salary since I need more additional income for some plans in the near future

• O company hiring process came first, although it took too long but they really showed effort to get me hired

• More potential to grow especially in career since they have plan to expand to Indonesia and I am their first engineer in Indonesia

The Job Hunting Journey

A little bit retrospective to see how I landed on a new job after applied to many job vacancies. I tried to tracked my job hunting data (recent 3-4 months) and made a data visualization as a summary.

So far I applied to 58 jobs (could be more) and surprisingly most of them are no answer or being ghosted by recruiter. 24 applications are failed, mostly on screening process and few of them failed after technical test or interview.

If I expand the data little bit especially the portion of which stage was contributed more to my hiring process. Ended up screening stage is the most contributed. Could be because no answer or simply they assumed I have no qualifications to went the next stage.

I think this is why we need to tweak our CV or resume to increase the chance of recruiter picked up our application. But portfolios, experiences, and tech stack could be the big factor for them to decide if we passed the screening or not

What is Next?

So yeah, this unemployed journal is complete for now. The next post, I will focus on regular content that is not related with the job hunting.

Cheers!

← Unemployed #10 - HTMX

It shows system information, memories, disk, CPU, and running processes, updated automatically every 5 seconds.

I’ll break down the code little bit in this post.

Stacks

• Go 1.23.2

• Htmx

• Tailwindcss

• Gopsutil

• Websocket

• Htmx websocket extension

HTTP Server

type HttpServer struct {
    subscriberMessageBuffer int
    Mux                     http.ServeMux
    subscribersMutex        sync.Mutex
    subscribers             map[*subscriber]struct{}
}

type subscriber struct {
    msgs chan []byte
}

It’s quite straightforward. HttpServer contains a http.ServeMux as http handler and subscribers for web socket broadcasting later. subscriber is simply has msgs channel for data update.

Since it only needs to serve a single HTML file, then it needs URL to show the page, and one URL for web socket connection.

func NewHttpServer() *HttpServer {
    s := &HttpServer{
        subscriberMessageBuffer: 10,
        subscribers:             make(map[*subscriber]struct{}),
    }

    s.Mux.Handle("/", http.FileServer(http.Dir("./views")))
    s.Mux.HandleFunc("/ws", s.subscribeHandler)
    return s
}

Web Socket Connection & Subscriber

Endpoint /ws will handling web socket connection and managing a subscriber. First it will initiate a new subscriber and added it to a map in the http server structure. Lock will be used to prevent race condition since we will use go routine later.

func (s *HttpServer) subscribeHandler(w http.ResponseWriter, r *http.Request) {
    err := s.subscribe(r.Context(), w, r)
    if err != nil {
        fmt.Println(err)
        return
    }
}

func (s *HttpServer) addSubscriber(subscriber *subscriber) {
    s.subscribersMutex.Lock()
    s.subscribers[subscriber] = struct{}{}
    s.subscribersMutex.Unlock()
    fmt.Println("subscriber added", subscriber)
}

Web socket is starting accept a connection and via loop, we will detect an incoming channel msgs from subscriber and write it to web socket.

func (s *HttpServer) subscribe(ctx context.Context, w http.ResponseWriter, r *http.Request) error {
    var c *websocket.Conn
    subscriber := &subscriber{
        msgs: make(chan []byte, s.subscriberMessageBuffer),
    }

    s.addSubscriber(subscriber)

    c, err := websocket.Accept(w, r, nil)
    if err != nil {
        return err
    }

    defer c.CloseNow()

    ctx = c.CloseRead(ctx)
    for {
        select {
        case msg := <-subscriber.msgs:
            ctx, cancel := context.WithTimeout(ctx, time.Second)
            defer cancel()
            err := c.Write(ctx, websocket.MessageText, msg)
            if err != nil {
                return err
            }
        case <-ctx.Done():
            return ctx.Err()
        }
    }
}

Auto Update

Auto update the system info data is handled by go routine. We will build a html response that will be sent via web socket and htmx will handle the update on the html side.

func main() {
    fmt.Println("Starting system monitor")
    s := server.NewHttpServer()

    go func(s *server.HttpServer) {
        for {
            hostStat, _ := host.Info()
            timestamp := time.Now().Format("2006-01-02 15:04:05")
            html := `
            <span hx-swap-oob="innerHTML:#data-timestamp">` + timestamp + `</span>
            <span hx-swap-oob="innerHTML:#system-hostname">` + hostStat.Hostname + `</span>
            <span hx-swap-oob="innerHTML:#system-os">` + hostStat.OS + `</span>
            `
            s.Broadcast([]byte(html))
            time.Sleep(time.Second * 5)
        }
    }(s)
    // ...
}

Syntax hx-swap-oob="innerHTML:#data-timestamp" in htmx is tell us that swap a component inside data-timestamp id in our HTML. All swapping mechanism will be the same for other system information components.

All swappable html components will be sent via Broadcast(msg) method and later will be sent via channel every 5 seconds.

func (s *HttpServer) Broadcast(msg []byte) {
    s.subscribersMutex.Lock()
    for subscriber := range s.subscribers {
        subscriber.msgs <- msg
    }
    s.subscribersMutex.Unlock()
}

The HTMX View

It’s plain HTML file and for Tailwindcss I simple used CDN for that

<script src="https://cdn.tailwindcss.com"></script>

Same idea for HTMX and web socket extension for using CDN

<script src="https://unpkg.com/htmx.org@2.0.3" integrity="sha384-0895/pl2MU10Hqc6jd4RvrthNlDiE9U1tWmX7WRESftEDRosgxNsQG/Ze9YMRzHq" crossorigin="anonymous"></script>
<script src="https://unpkg.com/htmx-ext-ws@2.0.1/ws.js"></script>

How to connect to the web socket?

The system monitor page is expected to receives all the data by web socket so I can set it from the main div container. Specify hx-ext=”ws” to tell HTMX for using web socket extension and ws-connect=”/ws” to tell web socket to connect via /ws URL.

<body class="bg-gray-700 text-white">
    <div class="container mx-auto p-8" hx-ext="ws" ws-connect="/ws">
   <!-- all the data -->
    </div>
</body>

Full Code

Here is the full version of the code https://github.com/didikz/gosysmon-web and you may want to play around with your own version.

Happy coding!

Decided to continue on my playground building a short url service using Go. The first one was how to implement templ in go as view template. It was interesting because we can separate the view components and made it reusable.

package components

templ Header() {
    <header class="bg-gray-800 py-4">
        <div class="container mx-auto px-4">
            <h1 class="text-2xl font-bold">Goshu</h1>
        </div>
    </header>
}

That component later could be called into other template by @components.Head(). As a person who familiar with Laravel Blade, separating components seems good idea.

Anyway the landing looks like this now:

Tried htmx on POST form got successfully implemented but I struggled little bit. It’s been a while I played around with html forms because for recent years mostly related with REST API and platform related development. Also I need to adapt on the htmx syntax including all the events, attributes, etc.

However I still couldn’t parse the JSON response on the views side once the form submitted successfully. Definitely will be my next research on that.

That’s it, have a good day!

← Unemployed #9 - Update On Job Hunting

Unemployed #37 - End of Search and Incoming Offers →

Sooooooooo, my contact on Glints already scheduled for 2nd interview round at 22th Oct next week. Took more than one month to get this progress after first interview. I already set my expectation to low on this, but still got hope because last week she said my next interview schedule will be arranged and just got the info today.

Let see how the interview goes next week. I’ll share the detail later.

Ed-Tech Company

Surprisingly this morning a recruiter company reached me for a senior role in one of SaaS company which has product for child education. Interesting because long back then I interested with educational topic. I tried to join several edtech company based in Indonesia but failed to get hired.

Definitely skill issue.

However this afternoon we discussed about the role. It is Singapore based company but has diverse team in Indonesia, Singapore, and Vietnam. Their main tech stack are Javascript and Go but there’s a PHP part that needs my expertise on that.

The discussion were about:

• Company detail, team size, product, and tech stack

• History of my experience

• Working preference and arrangement

• Previous and expected compensation benefit

• Hiring process

• Career expectation

Overall hiring stage would be:

• Technical test (home test)

• Panel technical interview with tech lead and engineering manager

• Final interview with CTO

• Offering

Once all sorted out and I agreed to continue, he will proposed me as their candidate and if all good then they will send the technical assessment. I’ll wait on that.

I also told him that I already committed to freelance project with my friend so in case I got hired, I need time to notify my client. He was ok with that.

Another good part is, he already has my profile. In case I failed to this company, he will find another company who might suitable with my expertise.

Freelance Update

No update yet. I already said that I would ready by next week and they still not sending any term or contract whatsoever.

Upwork

Naah, my connects is running low. I won’t use it until it necessary and worth to spend for potential projects.

That’s it. Have a great day, Folks!

← Unemployed #8 - An Unexpected Offer

Unemployed #10 - HTMX →

Still amazed me how one of them is actually my first hire as web developer back then when we’re still 5 peoples IT agency now becomes a project director. I remember he was a fresh graduate from college and I tested him with some basic CRUD using PHP. Couldn’t solve related with JQuery stuff but I decided to hired him.

It has good feeling knowing our work partner grows to a certain point, remembering who they were at the start. Also remember I was mentioning my high school intern then becoming a solid developer. Time flies.

System Performance Discussion

It was discussion about their web application performance and find what approach to solved it. The web application which supported client’s ERP system is actually working normally but during the peak usage, it started slow. The latency could be 5 minutes long which is unacceptable. The possibility is probably coming from slow queries due growing data in database or infra capacity issue.

Application hosted in a single server along side with the database server so we’re assumed that particular setup contributed to bottleneck during peak. Separate the database or even making it a master-replica is impossible right now because it was client’s IT department authority and the CTO won’t bother to make that requests. Just wasting a time.

We were discuss a lot regarding database issue. I suggested them to use profiling tools to track the bottleneck, otherwise it might ended up solving the wrong problem. They already had slow query and some profiling in Sentry but although it already been improved but slowness still occurred. I suggested to check database usage especially on threads and lock time. At Flip, I experienced many performance issues related with threads and lock time went high although the CPU and the memory was low. Indexing, connection pool, query strategy also been discussed.

Sudden Offer

Conversation led us to I said that I already resigned from Flip and looking for new opportunities. Suddenly he offered me a same position as my last position before I leaved the company, this time with slightly different title and scope of work. They’re facing some managerial issue in engineering department and needs me to help them sorted the problem.

This put a dilemma on me. I need a job but my priority is on IC position because I still wants to going deeper to technical expertise. I wrote some of my reasons on two separate blog posts why I choose IC as my current main goal:

• Career Retrospective: Individual Contributor or Engineering Management

• Finding the Archetype

Being managerial position is not my top of priority but it still valid option as career journey.

I answered that I still want to grind on technical position as IC though and I also mentioned that I already committed to a full time project with my ex coworker which could be took 3-4 months long. The other thing is I also still on hiring process on a commerce company based on Singapore which wants to expand their engineering team to Indonesia which still being my interesting choice.

The Choice

Yea, at the end I have to consider this offer so I said that I needs time to think and he was fine with it. But I offered back to him if I am available to part-time development job if needed. It could be the opportunity to “warm up” or observe the company condition before I take final decision whether want to join as managerial position or not.

So ya, I think for couple months forward I will focus on the upcoming project or waiting hiring result from Singapore company or even another interesting opportunity.

← Unemployed #7 - The Effort of Registering Vehicle Registration Number

Unemployed #9 - Update on Job Hunting →

I can do this by using we called “biro jasa” so I just need to sit tight and let them work, I’ll get the result later. But the downsides are:

• costly, might 2x or 3x from original cost

• I have no experience how to do administer this kind of activity in the future in case I need to do it by my self

Or other option I do this myself and will take less cost and experiencing it to gain knowledge how to administer vehicle tax and registration number. The downsides are:

• High effort, driving back and forth to multiple cities

• Longer time to complete, will took months

Then I decided to do it by myself. Took almost 6 months to finally got my new registered number but it was worth my effort.

I still wondering is it possible to make this process more efficient in this country? I understand that the vehicle needs to be physically there to manually checked by the verifier to prevent fraudulence. But all the paperwork was so long. It need at least your half a day to do the workflow like registration, verify the vehicle, paying taxes, and finally like need comeback 1 or 2 months to get the final result. Don’t forget to came early in the morning so you will get early queue.

If you don’t have any flexible working time, definitely needs a day off to do that or at least a half day off. Or like I said previously, “biro jasa” is there to help you but it will costly 2-3x. No problem if you have budget on that but most people don’t.

That’s it, have a great day!

← Unemployed #6 - Lost the Momentum

Unemployed #8 - An Unexpected Offer →

Or is it typical of starting a Monday?

However, I started a day with cutting my hair. Yes, I cut my hair by myself and I’ve been doing it since 2013. Definitely not as best as barber’s result but personally it’s quite satisfy me. After that I don’t really have spirit to do some works.

Several days ago I had been invited as guest writer on coderlegion.com and today I tried to do cross posting my new blog about Java Spring Boot to their platform. Apparently after posting the article, it still need to be moderated or reviewed. Fortunately it published successfully. You can check this link https://coderlegion.com/424/creating-restful-api-using-spring-boot-for-the-first-time.

I tried to work on my short URL project with digging on Go Template with goal creating landing page based on htmx and Tailwindcss. But my brain couldn’t able to think. I don’t know if it related to today’s weather was very hot (~38c) so my room was like a sauna.

Good News

The good news was my ex coworker in consulting company offered me a 3-4 months fulltime project. He said the stack are Go and Nestjs. Sounds fun!

The rate is not really high but enough to cover my monthly expense and also the opportunity to dig dive with Go & Nestjs, some stacks that I’ve been interested recently. I’ll take it as working and learning and the same time. I expected I will gain more expertise on those stacks.

Bismillah, wish me luck.

← Unemployed #5 - Good Friends

Unemployed #7 - The Effort of Registering Vehicle Registration Number →

Preparation

I am using Windows machine (Windows 11) here so it could be Windows specific

• Simply went to spring.io and check quick start page.

• Download and install JDK. I am using JDK 21 LTS

• Install Extension Pack for Java in VSCode

• Once JDK installed, check Windows Environment Variable. Need to add JAVA_HOME path from installed JDK in user variable. Check in terminal or powershel with java —version. If showed up then ready to go.

• Install Extension Pack for Java in VSCode. Then in settings.json add this config. If not, the package name would display a red mark although it’s fine.

"java.project.sourcePaths": [""]

Generate Spring Project

It’s good to know we don’t have initialize the project from scratch, we can use spring initializr.

We can define the project attributes here. My setup:

• Gradle

• Spring Boot default (3.3.4)

• Metadata default (demo)

• Packaging default (Jar)

• Java 17

• Spring Web as dependencies because need to build RESTful API

Generate the code and later just extract the downloaded code.

I also noticed when it opened with VSCode and the extension pack already installed, Gradle will automatically download the dependencies in background so later we don’t have to do it manually.

First Web Service API

The goal is I want to create a RESTful API that accept GET /greeting request and returned JSON response.

{
    "id": 1,
    "content": "Hello, World!"
}

To do that, I need to create resource representation class.

Spring uses https://github.com/FasterXML/jackson that automatically marshal the class into JSON

// /src/main/java/com/example/demo/Greeting.java
package com.example.demo;

public record Greeting(long id, String content) { }

Next, create a REST Controller to serve the request.

// /src/main/java/com/example/demo/GreetingController.java
package com.example.demo;

import java.util.concurrent.atomic.AtomicLong;

import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.GetMapping;

@RestController
public class GreetingController {
    private static final String template = "Hello %s!";
    private final AtomicLong counter = new AtomicLong();

    @GetMapping("/greeting")
    public Greeting greeting(@RequestParam(value = "name", defaultValue = "world") String name) {
        return new Greeting(counter.incrementAndGet(), String.format(template, name));
    }
}

• @RestController annotation to identify that this controller will handle HTTP requests as part of RESTful service.

• @GetMapping(“/greeting“) annotation it’s like a route, to ensure HTTP request to GET /greeting will be handled by Greeting function.

• @RequestParam() annotation for binds the value of the query string parameter name into the name parameter of the greeting() method. In this case if the value empty, default value world will be used.

• Finally it returned resource Greeting that we previously defined.

Running The Service

Go to Terminal and run this command to run the service

./gradlew bootRun

Or we can build the project and execute the .jar file

./gradlew build
java -jar build/libs/demo-0.0.1-SNAPSHOT.jar

My first attempt of running bootRun was failed. Later that was my mistake because my generated project was set to JDK 17 but I installed JDK 21 instead. Changing the language version in the toolchain was solved the problem.

java {
    toolchain {
        languageVersion = JavaLanguageVersion.of(21)
    }
}

After the service is running, the API now is accessible at http://localhost:8080/greeting

Tests

Lets digging more to the tests. It’s possible to test the controller using MockMvc package. In this case I want to test that Greeting controller is giving 200 HTTP Code and proper JSON format.

// src/tests/java/com/example/demo/GreetingControllerTest.java
package com.example.demo;

import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.jsonPath;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;

import org.junit.jupiter.api.Test;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.web.servlet.MockMvc;
import org.springframework.test.web.servlet.request.MockMvcRequestBuilders;
import org.springframework.http.MediaType;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc;

@SpringBootTest
@AutoConfigureMockMvc
public class GreetingControllerTest {
    @Autowired
    private MockMvc mvc;

    @Test
    public void getGreeting() throws Exception {
        mvc.perform(MockMvcRequestBuilders.get("/greeting").accept(MediaType.APPLICATION_JSON))
            .andExpect(status().isOk())
            .andExpect(jsonPath("$.id").isNumber())
            .andExpect(jsonPath("$.content").isString())
            .andExpect(jsonPath("$.content").value("Hello world!"));
    }

    @Test
    public void getGreetingWithName() throws Exception {
        mvc.perform(MockMvcRequestBuilders.get("/greeting?name=johndoe").accept(MediaType.APPLICATION_JSON))
            .andExpect(status().isOk())
            .andExpect(jsonPath("$.id").isNumber())
            .andExpect(jsonPath("$.content").isString())
            .andExpect(jsonPath("$.content").value("Hello johndoe!"));
    }
}

• @SpringBootTest annotation tells Spring Boot to look for a main configuration class (one with @SpringBootApplication, for instance) and use that to start a Spring application context.

• @AutoConfigureMockMvc annotation to auto inject MockMvc as the test component. It useful to only test the function layer instead of running full server.

• @Autowired annotation is like a automatic dependency injection for MockMvc into the class test

• @Test annotation to tell Spring that the particular function is a test case. My first attempt I missed this annotation and when I ran the tests, the test case wouldn’t be detected.

So by the code above you will know what the expectation of the test case, right?

Running the tests could be achieved at VSCode by navigate to Testing Icon on the left and click Play icon to run the tests.

Other approach is using command ./gradlew test

Code Repository

I already published the full code on this repository https://github.com/didikz/my-first-spring.

Happy building, Folks!

Today I went to a cafe to meet some of my good friends from my old company. Even though we’re working in different company, but we’re often worked together in a cafe (yes we’re mostly a remote worker). They’re super fun, always full of funny stories, but also always supports each other. I relieved could tell the story why I took this decision and many things.

The funny thing is, these friends of mine are like 9-10 years younger than me. My wife often told me as a joke If I hangout with them is like I’ve been their older brother who took care my younger brother and sister.

This friendship started a long a go when they were intern at my old workplace during their vocation high school era or being a junior developer after high school graduation. So ya growing together from a company who has less than 10 employees to almost 50 for years made the good relationship. I’m like super proud knowing how they growth from a high school kid into a professional engineer who worked at good tech company.

Sounds like a older brother, eh.

Anyway, my team on Flip sent me a farewell gift which arrived today. It’s like a culture in Flip if your co-worker getting married, or new baby born, or even left the company, they will give them a gift. And they asked me what I want for the gift. Spent couple hours to think, but at the end I asked a Redmi Watch to support my riding hobby.

That’s beautiful. Thank you for your kindness, Gans (.

So, do you have good friends from your work? Would like to hear your story.

←- Unemployed #4 - Riding Your Stress

Unemployed #6 - Lost the Momentum →

Even though I stopped many times to catch my breath, I finished it. Surprisingly didn’t feel much tired and felt my body more lighter!

Workout actually good for releasing stress. In my case is riding a bike because you can feel the air, sightseeing, or going to place where it’s hard to reach by car or motorcycle.

Didn’t much code today. Updating some SEO part on my website didiktrisusanto.dev based on SEO checker extension

Other than that, trying to following course on codecrafters.io for building your own Redis. This is interesting because I can learn about TCP connection and other system programming concept.

Ah about job application…

Nothing significant update. I applied to a remote job via bossjob yesterday and this morning the recruiter asked my latest resume so I gave it via platform chat feature. However I also tried to submit another proposals at Upwork. 1 proposal viewed by client, that was my best progress on Upwork for now.

I think that’s it. Have a good day!

←- Unemployed #3 - Runway

Unemployed #5 - Good Friends —>

Being unemployed mostly sucks right?

I know. Stop point at me, I took a long consideration before submitting my resignation letter

But at least to make it bearable, you have to consider these:

• Your available funds

• Your monthly expenses (bills, loans, foods, insurance)

• Your acceptable runway or how long you could live without a job in acceptable way

First, I’m feeling grateful I no longer have loans or installments so it will make huge difference on my monthly expenses calculation. Secondly, my wife is still have a job so at least for basic needs it would be covered. Last, I am living in a city where the prices still considered not to expensive.

Sorry my dear, I am “half” counting on you for now. Please be patient.

Now back to runway.

Excluding my wife’s salary, I break down my monthly expense and cut down some of it if possible. Then I add some emergency funds as a “unemployed” budget. So based on that, I have at least 8 - 10 months of runway. I could stretched it to 11 - 12 months with adding some budget from my other saving but I wouldn’t prefer any over budget here so let stick with 8 - 9 months max.

So 9 months will be my runway.

Hey I remember I possibly could take my funds on BPJS Ketenagakerjaan but I think I need to find reference about that. By looking the amount it probably could add 3 - 4 months additional runway.

Why not starting business

Bruh, I am pretty bad at this. But let see if I could find any interesting Idea for building digital business.

Retirement Saving

I’m having dilemma whether I keep add funds to my retirement saving or not while in this period. But first lets cut in half from original budget because I still need to prepare for retirement funds for my future.

If things go south, maybe will completely cut off the budget on this. We’ll see.

Other Update

No update yet on job applications today. I submitted 2 proposals in Upwork today regarding Laravel related project but I have no big expectation on that.

This morning someone reached me at email offering me to become guest writer on their blog platform. This is interesting opportunity to sharpen my writing skill and potential to reaching more audience. But I have no information where’s the blogging platform. They only said it similar to dev.to. Already replied asking for the detail.

I just published a blog about using Golang Migrate for manage database migration based on my experiment yesterday. Probably today will continue playing around with my short URL application.

That’s it, have a great day!

←- Unemployed #2 Trying Productive

Unemployed #4 - Riding Your Stress —>

There are several migration tools for Go but now let’s learn how to use Golang Migrate to manage migration in Go.

Preparation

• Database server in this post, I am using PostgreSQL

• For Windows, make sure it already installed Scoop

• Go

• Terminal

CLI Installation

Make sure Scoop already been installed and In your Windows terminal run this command to install migrate

scoop install migrate

Once it done, run command migrate in Windows terminal. If everything is good, then it will indicated CLI correctly installed.

For other OS installation can be check on the instruction detail https://github.com/golang-migrate/migrate/tree/master/cmd/migrate

Create Migration

We need a directory for storing the migration files. Create folder migrations in your Go project directory. You may create folder database/migrations or db/migrations depends on your preferred structure.

Go to your terminal and run the migrate command to start create migration files

 migrate create -ext sql -dir db/migrations -seq create_urls_table

• create option is to create migration files

• -ext sql is to create migration files with .sql extension

• -dir db/migrations path for migrations file will be stored

• -seq create_shortens_table will generate up/down migrations for your table schema file sequentially

You may adjust the command and arguments based on your project structure

Generated files should looks like this

Now you just need to write SQL statement to create / altering / drop the table schema. Here’s sample of mine for PostgreSQL:

// 000001_create_urls_table.up.sql
CREATE TABLE IF NOT EXISTS urls(
    id BIGSERIAL PRIMARY KEY,
    slug VARCHAR(50) NOT NULL UNIQUE,
    original_url VARCHAR(255) NOT NULL,
    user_id BIGINT NULL,
    visit_count INT DEFAULT 0 NULL,
    created_at timestamp DEFAULT current_timestamp,
    updated_at timestamp NULL,
    deleted_at timestamp NULL
);

// 000001_create_urls_table.down.sql
DROP TABLE IF EXISTS urls;

To maintain idempotent, always check if the schema exists or not

Running Migration

Migration files up/down already created meaning you will able to create the table or drop the table using recent migration files. Before running the migration, make sure you already setup the database because it will need database URL to run.

My URL for PostgreSQL is like below

postgres://postgres:securepassword@localhost:5432/mydatabase?sslmode=disable

// convention
driver://username:pass@host:port/dbname?sslmode=disable

Then we can now run up migration to create new table schema

migrate -database postgres://postgres:securepassword@localhost:5432/mydatabase?sslmode=disable -source file://db/migrations up

• -database tells migrate to use particular database server

• -source locates migration files path which is in db/migrations directory

• up tells migrate to run 000001_create_urls_table.up.sql

When we need to revert the migration, we can use down option to drop the table schema

migrate -database postgres://postgres:securepassword@localhost:5432/mydatabase?sslmode=disable -source file://db/migrations down

Other command can be referred to this https://github.com/golang-migrate/migrate/tree/master/cmd/migrate

Getting Database Value From YAML File

Our project often uses config file to store the database configuration. In my case, it stored on YAML file config.yml. In Migrate’s Github there’s example to do that.

First, install Python and pip. Then install pyyaml library for reading the yaml file.

 pip install pyyaml

Now we can modify our migration command to

migrate -database "$(cat config.yml | python -c "import yaml,sys; print(yaml.safe_load(sys.stdin)['database'])")" -source file://db/migrations up

It basically using same structure as previous command but now we read yaml file and parsing the database to get the URL value.

That’s it. Happy coding!

I had plan to stay productive this day. No particular plan for what I wanted to build or learn yet in the morning. But first thing in the morning after dropped off my wife to office, I went straight to bank because I got troubled for enabling their mobile application after changed my phone. This is become little bit urgent since my emergency fund is in that bank.

Turned out it was because my identity number registered in the bank already outdated, that’s why when I tried to input my identity number always got invalid error. 15 minutes later it sorted out and I was able to access the app again.

Drive straight back to home.

Visiting Job Application, Again

Really becoming daily routine as unemployed person, eh?

Yesterday I followed up to a recruiter the rejection reason, no replied yet. Already talked to friend who works there, he was little bit surprised about the rejection. Although I didn’t have a clue about the reason, I just assumed I failed because in first assessment I said NO if I willing to relocate. It because my friend said there are several peoples are working remotely in this town for the company so I guess it was ok to said no. But again it was just assumption, let’s skip it for now and wait for the official response (I doubt it).

Also I sent a reply to one of my job application which been responded the night before. Again, stated that it is WFO policy and asked if I willing to relocate. I said no and also gave them a reason why I choose remote working without loosing productivity. Very understandable for me if they choose to reject me on this one. I wonder why I applied to this particular vacancy knowing it’s not remote position hahaha.

Just got idea if I should write some like remote manifesto for my self where I explained why I prefer remote working

There are update for several my applications either my application already viewed or in CV review process. Let’s wait for several days.

Next, I checked on Upwork proposal. Seems already hired a person to do the job. Move on.

Working on Short URL Project

I have a fun project to build some kind of short URL application like bit.ly or other similar application. Built using Go and it’s still very raw, only covered index application and redirection function, no database integration yet. So I planned to integrate with PostgreSQL and DB migration tool today.

BTW, here’s the repository https://github.com/didikz/goshu

After prepared the database, I decided to use Golang Migrate as DB migration tool. Oh boy, spent couple hours to just make it usable and also integrated with PostgreSQL.

By afternoon I created the migration scripts successfully, query the data using sqlx, and redirecting the slug into the original URL. Will create a blog post for DB migration later.

←- Unemployed #1 - The Beginning

Unemployed #3 - Runway —>

This is not my first time

Yup, this is not my first time I am being unemployed or living without a job. Back in 2014, my first company which I worked with was shutdown and getting trouble to pay our several months salary. Beside the office drama, my saving was less than 5 million but it was no big deal since I was living at my parent’s house. I even bought an electric guitar so I’m not bored at home.

It was 3 - 4 months without full time job. I did a freelance gig to do some project with my college friend. Earned enough to increase my saving. I also tried to open a little cafe with my local friends but then I decided to left because I enjoyed coding more and let my friends continue the business.

Back then in my home town, working at home or freelancing was unusual. People started looking me as a person who had many leisure time. And being at home all the time felt like a burden, so I started to looking a job. Finally I got it in small software company called Ayowes! in Malang.

Preparing the Laptop

Acer Nitro is my only one daily laptop after I returned my work laptop to office. Last time I used for work was around 2022 before I joined Flip. After that it only being used for gaming and casual browsing. So today I decided to cleaning it and updating some development environment in it.

Cleaned dust in fans, re-pasta the processor and GPU made the laptop is more “relaxed” now. Previously the fans sounding like a plane taking off.

Checked PHP version still using version 8+, so it’s fine. Next, updating Go version because I want it to latest version (1.23.2). I tried to download the latest version and install from official page, it didn’t update the version automatically. Stackoverflow suggested to use Chocolatey instead,

choco upgrade golang -y

and it updated automatically.

Updating Some Online Portfolios

Because I no longer associated with my ex employers, I urgently need to update my online portfolios.

Updated my Github

Updated personal site at didiktrisusanto.dev

That’s it for now. I planned to enhance more for my personal site but still don’t have any idea how.

Submit First Upwork Proposal

I read it somewhere and my friend who actively doing some gig in Upwork said it hard to get a project. But hey, who knows I’m lucky? I started to choose the project and submit my first proposal to it.

I have no big expectation on it but let’s see.

Following Up Recruiters

I have several job prospects on my hands. This morning I got rejection email after first stage of hiring process from a tech company based on Yogyakarta. I have no clue what’s the rejection factor so I replied to ask for clarify which points that make me failed.

Why I asked this? This is important for me because I would want to know why I failed so I can get better on the other hiring process.

No response yet, but let’s see for couple days. If still no response? I’ll simply let it go.

Been following up other recruiter from Glints for a prospect at a social commerce company. Her last update was I made through the next step after first interview, but it was several weeks ago so I little bit worried. Fortunately there’s good news, still considered as passed to the next step and they will decide the schedule for next interviews.

Hopefully this is a good sign.

Unemployed #2 - Trying Productive —>

So I took this decision.

What I’ve Learned

Long time ago I dreamed building an amazing product and widely used by many peoples. Joining Flip was one of my best decision to achieve that goal. I didn’t really have many experience to build a SaaS product with a complex system and huge traffic before and at this company I’ve learned a lot how to do that.

Also learned a lot about

• Managing large legacy codebase yet still has quite good maintainability

• Learned Go

• Handling huge amount of traffic and billion value of transactions

• Handling production incidents

• Leading project or tech initiatives

• Many good technical insights from great engineers

• Making a good product that satisfies customers

• Insight about running organization for almost hundreds engineers

• Learned FE development (just a little)

• Many more…

The Legacy

Couldn’t tell any detail but I made 20++ RFCs for tech initiatives that almost 90% implemented successfully in my team. Also released 200++ merge requests into our main repository.

Hopefully my team don’t blame me for the messy codes I wrote.

What’s Next?

So I am still looking for the new opportunity especially for remote working. There’re several prospects but let's see.

On the other side I want to take a break for a moment, refreshing my mind, re-learning many things to reshaping my current expertise. Although many years of experiences, many rejections would still affecting my confident levels. At this point I am questioning myself whether this is an effect of the tech winter or just skill issues of mine.

One of my best friend in college offered me to help him build a digital product which we believed there’s potential on that. Not sure how, but I do have a plan to discuss it with him.

I am thinking about Upwork but looks very hard especially when you are new and have no reputation yet. Alternatively perhaps I will looking some coding gigs that could helped me grinding new tech. Not really fast pace but should be comfortable enough to gain experience safely while building portfolios.

That’s it, looking forward for the new chapter.

See you around, folks!

Surprisingly though, it was hard.

I know, seems skill issue or because didn't use it for years. But at the end, I made it to become a starter kit for future self or maybe you who need it.

Want to jump directly to the code? Here we go

https://github.com/didikz/laravel-inertia-react-starter

Stacks

I want to be specific with React so the main stacks are:

• Laravel (v11)

• Inertiajs

• React

• Tailwindcss

• Vite

Including suplementaries:

• Tailwindui

• Eslint

• Heroicons

Learning Journey

Basically after I install Laravel, I was only following all the steps on the Inertiajs documentation regarding setup server side and setup client side for React, but it turned out like playing with puzzle to get fit one by one.

So here are the missing pieces in Vite and React setup. Assumed that inertiajs for server already been installed.

1. Install React dependencies

npm install react react-dom @inertiajs/react @vitejs/plugin-react

2. Change .js extension file to .jsx

3. In /resources/js/app.jsx change js extention to .jsx in page resolver. So it will look like this

    resolve: name => {
       const pages = import.meta.glob('./Pages/**/*.jsx', { eager: true });
       return pages[`./Pages/${name}.jsx`];
    },
   

4. Go to vite.config.js and add plugin for react and laravel. Don't forget use .jsx extension becase I already changed that in the first place.

    import { defineConfig } from 'vite';
    import laravel from 'laravel-vite-plugin';
    import react from '@vitejs/plugin-react';
   
    export default defineConfig({
        plugins: [
            react(),
            laravel({
                input: ['resources/css/app.css', 'resources/js/app.jsx'],
                refresh: true,
            }),
        ],
    });
   

5. Create the Inertia root view file usually at /views/app.blade.php and include @vite directive. Once again change to .jsx extension on the js file.

6. It should be able to built and running the site but auto reload when I changed the script page it will got error.

   

7. The solution was simply add @viteReactRefresh directive before @vite in root Inertia file. It will inject some script that would allow us to hot reload when there is a change in our script.

    @viteReactRefresh
    @vite(['resources/js/app.jsx', 'resources/css/app.css'])
    @inertiaHead
   

8. That's it, the rest only need to create Pages and ready to go.

9. Running the service would be need two separate terminals

    # running the backend
    php artisan serve
   
    # Running the frontend
    npm run dev
   

Happy coding!

If you want to jump out directly to the code, check this final result on my Github Repository.

Load & Extract Data

import fs from 'fs';
const reader = fs.createReadStream(path, { encoding: "utf8" });
// handle errors
reader.on('error', (err) => console.error(`Error reading file: ${err}`));
// chunk the data stream and concat it into a single variable
let data = '';
reader.on('data', (chunk) => data += chunk);

The idea was reading a huge file contents and could be done without having huge resource consumption. I've found createReadStream function which able to read a file into a readable stream and I could chunk the stream for parsing the data later. My stupid idea was load the chunked data and concat it into single string variable.

Let see how it goes.

Parsing & Mapping Data

// at the end of the process, process the concatenated data and process it to map & sorting
reader.on('end', () => {
    const customers = data.split('\n')
                      .slice(1) // do not include header row
                      .filter(row => row !== '') // do not include empty row
                      .map(row => {
                         let splitted = [];
                         let current = '';
                         let inQuotes = false;
                         for (let char of row) {
                            if (char === '"') {
                                inQuotes = !inQuotes;
                            } else if (char === ',' && !inQuotes) {
                                 splitted.push(current);
                                 current = '';
                            } else {
                                 current += char;
                            }
                          }
                          splitted.push(current);
                          return splitted;
                       });
});

By the end of the stream now I will able to parsing the string from loaded chunks and determine values in each rows. I know there is csv-parse library that can be used but The challenge is only use standard or internal lib to solve the requirements. The logic are:

1. Split string by '\n' character

2. No need to parsing the header so skip it by using slice

3. No need to include empty row because it will counted as a row but has no data. Skip it using filter

4. Build customers array using map for parsing result

Assumed each column in a row is separated by comma so it will be easy, but turned out there are string that wrapped in double-quotes has comma in it.

7,EA4d384DfDbBf77,Darren,Peck,"Lester, Woodard and Mitchell",Lake Ana,Pitcairn Islands,(496)452-6181x3291,+1-247-266-0963x4995,tgates@cantrell.com,2021-08-24,https://www.le.com/

That unexpected data will ruin the parser so need additional logic to parser for detecting the string inside double-quotes.

Finally I already have customers array and I need to map it for city and customer count then sorted it by using sort function.

// map the data for city and total customers
let cities = new Map(); // 'city name': total customer
customers.forEach((val) => {
    if (cities.has(val[6])) {
        cities.set(val[6], cities.get(val[6]) + 1);
    } else {
        cities.set(val[6], 1);
    }
});

// sorted by the biggest customers first
const sorted = Array.from(cities).sort((a, b) => b[1] - a[1]);

console.log('rows count: ' + customers.length);
console.log(JSON.stringify(sorted));

First Result

It did great job when load 100.000 rows but failed at 1 million rows. When processing 1 million rows I got error like this below:

I think the problem was when I concatenated all the loaded strings into a single variable caused that error. Stored a huge string into a memory is a bad idea.

Improvement

Oke I need another approach and was thinking about async-await so I modified it into a function. Basically still using same createReadStream function but now I also tried using readline because it seems more proper way to read and iterate the data.

import fs from 'fs';
import readline from 'readline';

async function processCsvAwait(path) {
    // track the time
    console.time('processing_csv_time');

    const reader = fs.createReadStream(path, { encoding: "utf8" });
    const rl = readline.createInterface({
        input: reader,
        crlfDelay: Infinity
    });
    // the rest of codes
}

processCsvAwait('../data/customers-1000000.csv');

Then for extract and parsing the data should be like this below.

let count = 0;
let cities = new Map();
for await (const line of rl) {
   // process each row, excluding header row
   if (count > 1) { // skip header row
     // logic parsing still same like first implementation   
     // ...    
   }

   // mapping cities logic still same like first implementation
   // ...
   count++;
}

Improvement Result

After using improved method finally I was able to processed 1 mil rows with ~2 seconds time. Yeah that's a closed number with PHP result back then. Looks like still missing something for pushing the performance better but couldn't figure out where yet. Hopefully in the next #WeekendBuild.

That's it!

Reading CSV with PHP

Same concept for processing the CSV, we will load or open the given file and parse the csv data. Fortunately, PHP has standard lib for this called fgetcsv and we will use that function.

Load & Extract Data

Now opening the csv file in PHP is quite easy and can be achieved only using standard lib fopen

$f = fopen("../data/customers-1000000.csv", "r");
if ($f !== false) {
    // process the csv
} else {
    echo "Error opening file\n";
}

fopen received a file path string and "r" argument is a "read only" mode since we only need to read the files. This function will return resources when it successfully loaded or boolean false when it failed, so simple if else for error handling would do the job.

For extracting the rows, use fgetcsv and extract the row to an defined array to hold the value.

    $records = [];
    while (($csv = fgetcsv($file)) !== false) {
        $num = count($csv);
        $record = [];
        for ($c=0; $c < $num; $c++) {
            $record[] = $csv[$c];
        }
        $records[] = $record;
    }

This would work but when I tried to load 1 million rows, it got out of memory error:

PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes)

Not surprised at all and It's kind of expected because php array could took many resources especially dealing with the large data. I checked php.ini it stated that current memory_limit setting was only 128M.

Workaround on this is temporary increase the memory_limit on the runtime temporarily.

<?php
ini_set("memory_limit", "512MB")

So when we run the script, we also tell PHP to set memory limit to 512 MB before executing the rest of the process. Other way we could also change "memory_limit" in php.ini as for permanent change so all of php executions will be use that as default limit.

However this is not a good idea because increasing memory limit is indicating our process is not efficient and we have to avoid it.

Generators

Instead of building an array, PHP generators might be useful.

A generator allows you to write code that uses foreach to iterate over a set of data without needing to build an array in memory, which may cause you to exceed a memory limit, or require a considerable amount of processing time to generate. Instead, you can write a generator function, which is the same as a normal function, except that instead of returning once, a generator can yield as many times as it needs to in order to provide the values to be iterated over.

How we do that?

We can create a function to build the data using generators and called it later in our main function.

$f = fopen("../data/customers-1000000.csv", "r");
if ($f !== false) {
    $records = extractCsv($f);
} else {
    echo "Error opening file\n";
}

function extractCsv($file) {
    while (($csv = fgetcsv($file)) !== FALSE) {
        $num = count($csv);
        $record = [];
        for ($c=0; $c < $num; $c++) {
            $record[] = $csv[$c];
        }
        yield $record;
    }
}

Thats it!

Mapping and Sorting Data

Now since we're able to iterate the data we will mapping the customer data based on city and total customers. We could initiate an array to hold the map values, something like cityMap["jakarta"] = 100

$cityMap = [];
$rows = 0;
$records = extractCsv($f); // extract csv records and load it to iterator
foreach ($records as $key => $record) {
    $rows++;
    if ($key == 0) {
        continue;  // skip header row
    }

    // create array map for city and total customers
    if (array_key_exists($record[6], $cityMap)) {
        $cityMap[$record[6]]++;
    } else {
        $cityMap[$record[6]] = 1;
    }
 }

Previously in Go version we used bubble sort algorithm to manually sort the data from map, PHP has built in sort function called arsort to sort an array value in descending order but still maintain the keys correlation.

arsort($cityMap);
echo "sorted from most customers in the city: " . json_encode($cityMap) . PHP_EOL;

Getting Memory Usage and Processing Time

We need to initiate the memory usage and a timer before executing the main process and later we calculate it at end of the process.

<?php
// start profiling
$startMemory = memory_get_usage();
$timeStart = microtime(true); 

// starting main processes
// ...
// end of the process
$timeEnd = microtime(true);
$endMemory = memory_get_usage();

echo sprintf("processing time: %f (s)", ($timeEnd - $timeStart))  . PHP_EOL;
echo sprintf("memory usage: %f (Mb)", ($endMemory - $startMemory) / 1024 / 1024)  . PHP_EOL;

The Result

Running php application is simply to run with

php index.php

The result for loading and processing 1 million rows of csv was:

• Processing time: ~2-3s

• Memory usage: 0.030350 (Mb)

Not bad. Performance wise, Go is better since it's a compiled language meanwhile PHP is not.

Full source code can be checked on my Github: https://github.com/didikz/csv-processing/tree/main/php

Lesson Learned

Handling large data in PHP is quite tricky and we need to be aware about the memory usage especially when dealing with the arrays. We could increase memory_limit as "urgent" solution especially when there's an issue in production but don't do that. Optimize the application instead.