After setting up Elasticsearch locally I needed to seed it with some dummy data.

I came across this article that explains all the steps in details, including populating the postgres with dummy data to begin with.

I did not want to populate the dummy data since I already had some data in postgres.

But next step, to export the data into json format proved difficult because I use dockerized postgres.

The > export.json would not work for me, because psql ran with pager, and would wait at -- More -- prompt. I saw a first couple of lines in export.json and -- More -- at the end.

I tried some options like turning the pager off from psqlrc - which works locally, but I could not get it to work in docker.

Then I decided to break it into separate steps.

First, get the data into json format:

I entered docker exec psql and in interactive mode ran the following:

1
2
3
4

\t
\a
\o export.json
select json_agg(t) FROM (SELECT * from table) t;``

Now I had export.json inside the docker container. So docker exec -it my-pgsql bash

I found the file in root folder, so I moved it to location that is accessible from outside.

cp /export.json /var/lib/postgresql/data/

Now I got it out using docker cp command

docker cp <container_id>:/var/lib/postgresql/data/export.json ./raw_export.json

See this on how to get the correct volume path

I had to format it so that elasticsearch would accept it

cat raw_export.json | sed 's/^\[//' | sed 's/^[[:blank:]]//' | sed 's/[[:blank:]]$//' | sed 's/,$//' | sed 's/]$//' > export.json

Rest of the steps were similar to the original article. I had to change format-json.js per my database structure.

The original article is from late 2018. Both elasticsearch and elasticdump have changed since then.

I had problem with https for local elasticsearch server. I bypassed it via setting up the environment variable

NODE_TLS_REJECT_UNAUTHORIZED=0 node_modules/elasticdump/bin/elasticdump --input=formatted.json --output=https://dev:elastic_dev@localhost:9200/

See the documentation here

That was it.

I was able to search via Kibana 🎉