SpreadIT 2016 – behind the scenes

30 November 20161 December 2016 Danielspreaditpl1 Comment

SpreadIT 2016 is over! I would like to use this occasion to share my thoughts and show how much effort is needed to create free of charge, technical conference with more than 500 attendees. As some people can find parts of this article offending, please note that this is only my point of view.

A little bit of current situation

First of all, we are not connected to any company and never were. SpreadIT is totally community-based event. For sure, we are very grateful to our sponsors, because without them we would not be able to do this. What we do every year is to create an offer and send it to companies. Every year we have a bit more confidence that we will have enough money. If we’ve possibility to save some extra money – we use it for next year edition (In fact, we need to leave some money as we must print some posters etc. before we receive first money for the next edition).

At the moment conference “management” is just group of 7 people. And these people meet each other regularly on some online meetings. Besides, we have direct communication with each other on slack channel. Everyone of us is 100% responsible for some part of conference which means that it lies on your shoulders to deliver that part of event in time. Everyone can create own “team” to gain some help of course. But… there is not too many people out there who wants to use their free time to write emails, Facebook posts, tweets, do some coding, graphics etc. We are extremely happy when someone new join us (THANK YOU GUYS!).

drink

365 days of preparations

I wrote above that everyone is responsible for some particular tasks. Let’s look at them more deeply:

Program – preparation include asking some well-known trainers with a lot of conference experience. We do this because we want new faces every edition. Then there is C4P – we need to look at them, often there is need to discuss particular topics if they (and creators) meet our requirements. Also we give possibility to do C4P by gold and platinum sponsors. But they also have to meet all requirements, and sometimes it could be somehow challenging- in one hand they are our sponsors, they give us money, in other hand – our priority is to establish best possible conference schedule. You can’t invite too much people, you can’t have less than some reasonable number,
Sponsors/Finances – we need money. Also we want to do some networking between our attendees and companies. Every company have some specific “culture”, sometimes there are specific requirements, sometimes there is a lot of questions. Imagine that you have to write a lot of emails to some official company mailing addresses (with a lot of redirections) to get sponsors. This part also include “keeping hand on money”, because at the same time rest of SpreadIT team ask you for money for their targets. That’s on daily basis for head of finance 🙂
Promotion – Facebook, Twitter, Media patrons, writing news, checking grammar, trying to post something concrete and interesting from time to time… Then preparing posters, advertisements, graphics, ids, folders, stickers etc.
IT – We have our webpage (which is all the time “under development” :)) – sometimes there are problems, we also have new ideas, requirements, needs…
Logistic – Someone have to find place for conference, prepare coffee breaks, check audio, mics, adapters for macs or specific ports, order pizzas (find pizzeria which will do 100 pizzas in reasonable time :)), find places for sponsors on conference day, do “shopping”,
Strategic contacts – (new role which was introduced first time this year). Mostly discussions with particular contacts, but we found out that this role takes some responsibility which I described under logistics. Also help with charity action which we do this year
General coordinator – during preparation phase coordinator is responsible for conflict solving and scheduling meetings, but most important coordinator remove all roadblocks, so everyone can do “their job”. It seems like the most easy role, but in fact this is very serious, because responsibility that coordinator takes is quite big.

knight

“Far side” of the conference

There are some behaviors that makes me sad:

In first 3 editions there were some people who just register to receive “free t-shirt”. They come in on registration, get their starter pack and then go back to home. Fortunately this year because of charity action we didn’t have this problem anymore. However, there were two or three comments like “we don’t like SpreadIT anymore because we didn’t receive free t-shirt”. I don’t want that kind of people on SpreadIT. Seriously.

A lot of criticism. I understand comments about cold pizza, about technical problems, about topic which didn’t meet participant requirements. I just don’t understand pure hate because we can’t afford something or we didn’t do something in particular way. Unfortunately we have small budget and this is always choice how to spent money. Don’t understand me wrong- We want all kinds of suggestions- we need to make a lot of hard decisions and its often about spending money this way or another. I just don’t get why some people have such hate for everything which is not “their way”. Live and let live. Really.

Registered people who don’t show up. It’s quite interesting, about 30% of registered people just didn’t show up on conference day. It’s fine if someone have to change their plans, many things can happen, but 150 people? Please understand that we have to prepare every single starter-pack, registration points, order pizza and many more. I know that you don’t pay for it so “just in case I will register”. Please have some empathy. 150 people is about 2hrs of additional work the day before conference for 10 people.

Bright side of the conference

There are still many reasons why we organize SpreadIT. For me it’s about some networking, about challenges, about growing up and, to be honest, about making my dreams come true. It’s a hard work, but on that day when I can stand in front of these 500+ people and say “we did it!”-it’s priceless. Especially if we did it good.

Call4Proposals and Call4Orgs 🙂

selfie

If you would like to speak at SpreadIT 2017 you can either wait for our regular Call4Proposals or contact me at daniel.pokusa(at)spreadit.pl 🙂

If you want to join us, meet cool people and make something great, do not hesitate and contact me or any other of SpreadIT committee member (mails here)!

See you next year!

New domain, new hosting

26 November 2016 Danieldomain, hostingLeave a comment

I changed domain to software-empathy.pl and also, what is more important, I changed the hosting provider. Unfortunately, last one was quite annoying for me (downtimes every few days and very poor response times – especially within admin panel).

I hope that new one will be better 😉

Asynchronous call using CompletableFuture

10 November 2016 Danielconcurrent, java, programmingLeave a comment

Intro

Imagine that you have to use external systems to retrieve data or you deal with communication in distributed architecture (communication between modules within application). In this article I would like to tackle one possible problem, but please note that this approach can be used in a many different scenarios.

Please note that I don’t want to provide another “CompletableFuture guide”. If you are interested in specific options and possibilities of CompletableFuture I recommend you great article by Tomasz Nurkiewicz.

Problem

We would like to write simple application that will provide us best possible exchange rate from public exchange rate systems (let’s assume that bigger exchange rate value is better as we want to sell). What we want is to convert from our “local” currency (in my example it will be PLN) to selected one (using simple rest call). Response should inform us only about highest rate.

So if I run my application on localhost:8080 and I would like to check lowest exchange rate for GBP then it can looks like:

╭─dpokusa@venture ~  
╰─➤ http :8080/GBP
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
Date: Sat, 05 Nov 2016 17:20:25 GMT
Transfer-Encoding: chunked
X-Application-Context: application:8081

4.8631

╭─dpokusa@venture ~

╰─➤ http :8080/GBP

HTTP/1.1 200 OK

Content-Type: application/json;charset=UTF-8

Date: Sat, 05 Nov 2016 17:20:25 GMT

Transfer-Encoding: chunked

X-Application-Context: application:8081

4.8631

(In example above I used httpie. I also recommend postman if you prefer more GUI-oriented tools :))

Another example (USD):

╭─dpokusa@venture ~  
╰─➤ http :8080/USD
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
Date: Sat, 05 Nov 2016 17:25:12 GMT
Transfer-Encoding: chunked
X-Application-Context: application:8081

3.8933

╭─dpokusa@venture ~

╰─➤ http :8080/USD

HTTP/1.1 200 OK

Content-Type: application/json;charset=UTF-8

Date: Sat, 05 Nov 2016 17:25:12 GMT

Transfer-Encoding: chunked

X-Application-Context: application:8081

3.8933

In our example we will check 3 external services for exchange rates:

NBP (Polish National Bank) – http://api.nbp.pl/en.html
Fixer – http://fixer.io
Yahoo Finance API

Concept

First of all as you can imagine we will make a few external calls to described above systems (every call will take some time). Best option is to call asynchronously to every exchange rate API and after receiving all rates just check which one is the highest and finally make a response.

We don’t want to make every request synchronously, because we will increase response time from our application.

Solution

Basic solution

Let’s start with some general interface for every API call:

interface ExchangeRate {
    BigDecimal retrieve();
}

interface ExchangeRate {

BigDecimal retrieve();

}

Then let’s provide some implementation for one of calls:

class NBPExchangeRate implements ExchangeRate {

    private static final Logger log = LoggerFactory.getLogger(NBPExchangeRate.class);
    private static final String NBP_API_CURRENCY_RATE_PATH = "http://api.nbp.pl/api/exchangerates/rates/a/%s?format=json";

    private final String url;
    private final String currency;

    NBPExchangeRate(String currency) {
        this.currency = currency;
        this.url = StringFormatter.format(NBP_API_CURRENCY_RATE_PATH, currency).getValue();
    }

    @Override
    public BigDecimal retrieve() {
        RestTemplate restTemplate = new RestTemplate();
        NBPApiResponse result = restTemplate.getForObject(url, NBPApiResponse.class);
        BigDecimal currentRate = result.getRates()[0].getMid();
        log.info("Current rate for {} is {}", currency, currentRate);
        return currentRate;
    }

    private static class NBPApiResponse {

        private String table;
        private String currency;
        private String code;
        private Rates[] rates;

        // getters&setters here
    }

    private static class Rates {

        private String no;
        private BigDecimal mid;

        @JsonDeserialize(using = LocalDateDeserializer.class)
        private LocalDate effectiveDate;

        // getters&setters here
    }
}

class NBPExchangeRate implements ExchangeRate {

private static final Logger log = LoggerFactory.getLogger(NBPExchangeRate.class);

private static final String NBP_API_CURRENCY_RATE_PATH = "http://api.nbp.pl/api/exchangerates/rates/a/%s?format=json";

private final String url;

private final String currency;

NBPExchangeRate(String currency) {

this.currency = currency;

this.url = StringFormatter.format(NBP_API_CURRENCY_RATE_PATH, currency).getValue();

}

@Override

public BigDecimal retrieve() {

RestTemplate restTemplate = new RestTemplate();

NBPApiResponse result = restTemplate.getForObject(url, NBPApiResponse.class);

BigDecimal currentRate = result.getRates()[0].getMid();

log.info("Current rate for {} is {}", currency, currentRate);

return currentRate;

}

private static class NBPApiResponse {

private String table;

private String currency;

private String code;

private Rates[] rates;

// getters&setters here

}

private static class Rates {

private String no;

private BigDecimal mid;

@JsonDeserialize(using = LocalDateDeserializer.class)

private LocalDate effectiveDate;

// getters&setters here

}

As you can see there, it’s very simple standard code. I use Spring’s RestTemplate to prepare and make external call. What’s important here – so far we don’t use any CompletableFuture so we can test this kind of classes as usual.

What I want to show is that you can provide functionality without thinking about async invocations. You can add this after finishing basic implementation.

Lets go async

Let’s look what we can achieve by using CompletableFuture . This class provides you possibility of creating object that will do some operations “in background” (We will talk about executors which do this operations later in this article). This gives you possibility to create non blocking operations (such as external calls) for future usage. Once you want to use information from CompletableFuture it start blocking until Future is completed. However, if operation is already completed then it will not block thread at all.

This can help you in a couple of scenarios:

You want to do some expensive operation and use it later. Then you can create CompletableFuture at first, make other required operations and then use completed Future . In case it’s not completed your app will just wait for it
You want to do some expensive operations concurrently and then combine them in some way (this is what we want, and this is most popular scenario)

How do you create CompletableFuture ? The easiest way is to use static factory methods provided by this class. For example: CompletableFuture.supplyAsync(Supplier<u> supplier)</u> . Supplier class is a @FunctionalInterface so you can use lambdas 🙂

Lets try then to provide some Factory for creating our asynchronous calls. This is not the best option, however in this case it can be a good starting point:

@Component
public class ExchangeRateFactory {

    public CompletableFuture<BigDecimal> nbpRateFor(String currency) {
        return prepareFuture(new NBPExchangeRate(currency));
    }

    public CompletableFuture<BigDecimal> fixerRateFor(String currency) {
        return prepareFuture(new FixerExchangeRate(currency));
    }

    public CompletableFuture<BigDecimal> yahooRateFor(String currency) {
        return prepareFuture(new YahooFinanceExchangeRate(currency));
    }

    private CompletableFuture<BigDecimal> prepareFuture(ExchangeRate exchangeRate) {
        return CompletableFuture
                .supplyAsync(() -> exchangeRate.retrieve())
                .exceptionally(t -> BigDecimal.ZERO);
    }
}

@Component

public class ExchangeRateFactory {

public CompletableFuture<BigDecimal> nbpRateFor(String currency) {

return prepareFuture(new NBPExchangeRate(currency));

}

public CompletableFuture<BigDecimal> fixerRateFor(String currency) {

return prepareFuture(new FixerExchangeRate(currency));

}

public CompletableFuture<BigDecimal> yahooRateFor(String currency) {

return prepareFuture(new YahooFinanceExchangeRate(currency));

}

private CompletableFuture<BigDecimal> prepareFuture(ExchangeRate exchangeRate) {

return CompletableFuture

.supplyAsync(() -> exchangeRate.retrieve())

.exceptionally(t -> BigDecimal.ZERO);

}

I also added exceptionally here for each created CompletableFuture – this means that in case that some exception will stop completion of particular Future then provided method will return value as a result. Receiving Throwable t gives you possibility to adjust result to situation. In this simple example it will return 0.

Finally. we are ready to use it!

@RestController
public class BestCurrencyController {

    private static final Logger log = LoggerFactory.getLogger(BestCurrencyController.class);

    @Autowired
    private ExchangeRateFactory exchangeRateFactory;

    @RequestMapping(value = "/{currency}", method = RequestMethod.GET)
    public BigDecimal bestCurrencyRate(@PathVariable String currency) {

        CompletableFuture<BigDecimal> nbpRate = exchangeRateFactory.nbpRateFor(currency);
        CompletableFuture<BigDecimal> fixerRate = exchangeRateFactory.fixerRateFor(currency);
        CompletableFuture<BigDecimal> yahooRate = exchangeRateFactory.yahooRateFor(currency);

        CompletableFuture<BigDecimal> best = 
                nbpRate
                .thenCombineAsync(fixerRate, (nbp, fixer) -> selectBest(nbp, fixer))
                .thenCombineAsync(yahooRate, (combined, yahoo) -> selectBest(combined, yahoo));

        try {
            return best.get(1000, TimeUnit.MILLISECONDS);
        } catch (InterruptedException | ExecutionException | TimeoutException e) {
            log.error("Can't retrieve best currency rate ", e);
            return BigDecimal.ZERO;
        }
    }

    private BigDecimal selectBest(BigDecimal first, BigDecimal second) {
        log.info("selecting best between " + first + " and " + second);
        return first.max(second);
    }
}

@RestController

public class BestCurrencyController {

private static final Logger log = LoggerFactory.getLogger(BestCurrencyController.class);

@Autowired

private ExchangeRateFactory exchangeRateFactory;

@RequestMapping(value = "/{currency}", method = RequestMethod.GET)

public BigDecimal bestCurrencyRate(@PathVariable String currency) {

CompletableFuture<BigDecimal> nbpRate = exchangeRateFactory.nbpRateFor(currency);

CompletableFuture<BigDecimal> fixerRate = exchangeRateFactory.fixerRateFor(currency);

CompletableFuture<BigDecimal> yahooRate = exchangeRateFactory.yahooRateFor(currency);

CompletableFuture<BigDecimal> best =

nbpRate

.thenCombineAsync(fixerRate, (nbp, fixer) -> selectBest(nbp, fixer))

.thenCombineAsync(yahooRate, (combined, yahoo) -> selectBest(combined, yahoo));

try {

return best.get(1000, TimeUnit.MILLISECONDS);

} catch (InterruptedException | ExecutionException | TimeoutException e) {

log.error("Can't retrieve best currency rate ", e);

return BigDecimal.ZERO;

}

private BigDecimal selectBest(BigDecimal first, BigDecimal second) {

log.info("selecting best between " + first + " and " + second);

return first.max(second);

}

Completable Futures are being created in lines 12-14, then (lines 16-19) we take nbpRate as our basic value, combine result with fixerRate and select better option (higher), then again we combine this result with yahooRate . This will create another CompletableFuture – let’s call it best.

Finally, as we don’t have anything more to do we use get() method on our Future . This method as I wrote before is blocking- so we will wait until our Futures will complete and we will have our solution. Then we will just return it as a result.

Now you can see why I said that previous fabric its not our best option. If you have to add new service (and its very likely in our problem) then you will have to add here new lines and implementation which breaks Open/Close principle. I use this approach for sake of simplicity of explanation.

Executors

You can run CompletableFuture by using default Executor (as in all examples in this article) what can be fine for simple solutions. But for real world complex requirements this probably will be not effective and can lead to serious problems. If you want to learn more about default executors, Oleg Schelajev did great article about choosing right executor for your tasks.

However, if you are using spring you probably should provide TaskExecutor which has similar interface as java.util.concurrent.Executor , but Spring provides a few nice and configurable implementations of it.

Take an example:

@Configuration
public class ExecutorConfiguration {

    private static int CORE_POOL_SIZE = 2;
    private static int MAX_POOL_SIZE = 10;

    @Bean
    public TaskExecutor exchangeRateTaskExecutor() {
        ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
        ThreadFactory threadFactory = new CustomizableThreadFactory("ExchangeRateTaskExecutor-");
        executor.setThreadFacstory(threadFactory);
        executor.setCorePoolSize(CORE_POOL_SIZE);
        executor.setMaxPoolSize(MAX_POOL_SIZE);
        TaskExecutor taskExecutor = new ConcurrentTaskExecutor(executor);
        executor.initialize();
        return taskExecutor;
    }
}

@Configuration

public class ExecutorConfiguration {

private static int CORE_POOL_SIZE = 2;

private static int MAX_POOL_SIZE = 10;

@Bean

public TaskExecutor exchangeRateTaskExecutor() {

ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();

ThreadFactory threadFactory = new CustomizableThreadFactory("ExchangeRateTaskExecutor-");

executor.setThreadFacstory(threadFactory);

executor.setCorePoolSize(CORE_POOL_SIZE);

executor.setMaxPoolSize(MAX_POOL_SIZE);

TaskExecutor taskExecutor = new ConcurrentTaskExecutor(executor);

executor.initialize();

return taskExecutor;

}

This configuration provides ThreadPoolTaskExecutor which uses CustomizableThreadFactory . One can set max pool size (max number of threads that will be created), core pool size (base number of thread that should be in use) and of course more options.

Don’t use default executor. If you use spring, just go with your own configured TaskExecutor . If you use JEE or other frameworks pick up one of available executors in java.util.concurrent package. Concurrent programming is very challenging so if you can keep something under control then definitely you should do it 🙂

To use particular executor just use second implementation of supplyAsync method:

CompletableFuture
                .supplyAsync(() -> exchangeRate.retrieve(), executor)
                .exceptionally(t -> BigDecimal.ZERO);

CompletableFuture

.supplyAsync(() -> exchangeRate.retrieve(), executor)

.exceptionally(t -> BigDecimal.ZERO);

Afterwords

There are more problems around this topic, for example, you need to figure out how to deal with situation when external service is down. We also need to consider how to test our solution and how to simplify future maintenance. There is always a need of providing a good balance- we should always provide sufficient, smallest and simplest possible solution- it’s not easy task to do though 🙂

Most important criteria that should be considered before providing final solution:

Reliability
Resilience
Maintainability
Testability
Performance

SpreadIT 2016 – What’s new at the 4th edition?

5 November 2016 DanielspreaditplLeave a comment

Unfortunately I don’t have too much time recently to write new posts. One of the reasons: SpreadIT 2016.

As I’m part of this great community event I want to share some more info about this year edition:

We decided to give up workshops
Finally we change “Soft talks” path name to “Business and Personal Development”
We want SpreadIT 2016 to follow simple rule: bigger, better, louder 🙂
Our Team changed – we’ve got new people on board
We will have more than 20 presentations (5 rooms parallel – look here)
There will be more than 20 great professional speakers (Look here)
We decided that we will no longer include t-shirts within starterpack. We can’t afford t-shirt for every attendee, and don’t want to (as in previous editions) give them only for “first X registered attendees”. So this year if one want to get SpreadIT custom and limited t-shirt then will need to help sick kids. We choose “Mam Marzenie” charity foundation as beneficent of special charity auction (done by “Allegro Charytatywni”). Every penny spend on our auctions goes directly to foundation and will help to make a sick kid dream come true! You can find these auctions here. Its just 15 PLN for each t-shirt (Of course you can be more generous!)
We want to start one new “culture thing” related to SpreadIT. But for now it is a secret 🙂

Last year there was 500+ attendees, we served more than 70 pizzas (something about 560 large slices :)) and what is important – we are just a bunch of passionate people who want to do something good for local community. SpreadIT is not directly created by any company (of course we are very grateful to our sponsors!).

SpreadIT will be held on Silesian University of Technology in Gliwice on 19.11.2016. For more info visit our site: https://spreadit.pl/en. Remember that our event is totally free. Just go to our site and register for the event (And remember about our cool t-shirts!)

See you there!

Software Empathy

Personal thoughts about coding and software architecture

Month: November 2016