In my opinion the course is designed very poorly. Firstly, I was expecting the Capstone to cover different aspects of the rest Scala courses more or less equally. But in fact this course is 80% about math and java performance tuning and only 20% about very basic Scala. Secondly, due to the poorly designed given skeleton methods it is very difficult and cumbersome to write a performance effective solution. In order to generate real images I had to write two implementations - one for the grader give me 10/10 and another one that really works and is capable to generate images effectively. Of course I had to get rid of all the given method signatures and reassemble the pieces in different way. E.g. performing more computation in Spark, combining several steps of computation in one, preferring stream style processing with less memory allocation etc. Actually I have found myself spending more time fighting with the given skeleton and the grader, rather than writing a workable solution.

Also, if I were designing this course I would rather go from the top to bottom when slicing the implementation into layers, steps and weeks. So the first step would be to implement a UI using mocks, then replace some mocks with the real code and so on down to the Spark level, reading the CSVs and proving the real data for the solution. This way the student would clearly understand how the methods being implemented are used, giving him much more input for creating a good design, rather than just blindly implementing some methods that _somehow_ will be used in the next week etc.

The course's introductory description, "In the final capstone project you will apply the skills you learned by building a large data-intensive application using real-world data." is quite misleading. The fact that this course is cataloged under "Data Science > Data Analysis" also suggests that the course's project is data-centric, but that's not the case at all. This course requires very little data manipulation (basically only the first milestone) and no data analysis at all. The project is for the most part a "Slippy Map" implementation in Scala. Suggesting that this application has anything to do with data science to appeal to customers is down dishonest.

But even if you're forgiving about the absence of data science and you're satisfied with Scala only and think that you can use the final product of your labor to showcase your skills to employers, I'm afraid that it wouldn't be very helpful.

The "fill-in the missing code" grading model (all methods signatures are pre-defined and you have to complete them) robs the student from the opportunity to practice or showcase any object-oriented / functional-programming design skills. In my experience, that's what employers want to see. Can the candidate design code that is easy to change? Does the candidate's code follow S.O.L.I.D. principles?

For the employers that are more algorithm-centric in the interviews, I think that is unlikely that they will ask you to implement algorithms taken from Wikipedia. They will usually give you a problem and ask you to either design an algorithm or find the proper algorithm to solve a problem on your own.

The instructor also suggests that Spark could be used, but that's untrue. Even though the library dependency is in the SBT file, you can't actually use it given that the template code wasn't designed for this purpose (large collections are unnecessarily kept in memory and you can't change the design without making it incompatible with the grader) and, again, you would only use Spark on the first milestone.

Unless you're interested in web maps, I don't think that this project will be very useful for you.

Regarding grading, I managed to make every single milestone pass the grader's tests, but that's not an easy task given that the grader messages are quite cryptic and don't follow TDD best practices (tests should be "executable specification", they should document the requirements and should be readable).

Many times a submission wouldn't pass the grader because of small differences in the parameters you have chosen in your implementation. The grader messages aren't very useful and given that you don't have access to the grader's source code, you have to resort to trial-and-error, which can be very frustrating.

They even hide crash and timeout messages, so your submission might fail and you don't even know which one is to blame. All you get is a "0/10" score and no clues on why it happened.

I truly recommend anyone who wants to showcase Scala work, to create an application from scratch. Even if you're not bothered by the cons that I mentioned, I'm pretty sure that soon enough (I took the first iteration) you will find people sharing their solutions on GitHub, which makes the certificate pretty much meaningless.

In this course, you are asked to apply your Scala knowledge to a more practical project.

However, I found the tasks rather confusing and lost track of the actual purpose of the application. In the end I just implemented everything function by function. So it felt more like in exercise in understanding the goals of the course instead of a Scala training.

If the goal is to develop an application that feels a bit more real-world, it would be good to have more vivid explanations of the business logic, for example in videos and not just as text.

All the exercise are poor explained and It's need a hard math background. It's not a scala or functional course, It's a math exercise problem with scala.

Let's start with cons. Not often, but one have confusing exercise explanation(method docs contradict with assigment text), sometimes explanation is unclear in general. Too much geo math and in general course exercises look like "realize some formulae" definition. Grading tests output basically useless and it's sad. Tests must be the main point that explains proper way to right implementation and at some point being spec explanation.

But some pros for me exist. Geospatial is not one of my biggest-fav-domain, but it was interesting to dive into the new fields. The next thing is the oportunity to work with some kind of real world data processing scala application, it was quite interesting.

I give it four stars and a half. At the end my feelings tell me that I rather like this course than not. Yes, it has some certain rough edges, but this course help you to level up your scala and spark skills and to fill one's hand.

In the end I want to give some advices. If grader falls with out of memory, try to optimize extraction code. Spark is not always the right tool for all cases. Also, Spark has awesome web ui where you could watch what happens and where is main bottlenecks. Write your own tests... even simple test can help to find out bugs that you could think is not expected. That's all. Thanks for the good course.

The course was supposed be a practical application of everything we've learned in all previous courses. I was not. Instead of learning functional design or applying big data analysis framework everything just worked with simple ".par" after large collections. I don't remember myself optimising memory usage or reducing shuffling with spark. What I do remember are hours of fiddling with floating point precision trying to figure out why a test fails with "30 is not equal to 30.06". Moreover, tests were just not good enough: even after I've passed week 1, I had to come back to that code when tests in subsequent weeks were failing. Lastly, complexity of the weeks is badly balanced. It took me three weeks to complete Week 3, but I've finished weeks 4-6 in about 2 hours.

Good course, some of the assignments could have been more explicit with expectations in cases where specific implementation details matter but the forums were helpful in that regard anyways.

I've read a lot of reviews before me and see that they mentioned several cons: off-topic programming assignment, imbalanced difficulty distribution and instable grader, vague feedback information.

For the few points mentioned above, I'd say I've experienced all of them indeed, it's true.

But for a project that's to be called "real", you're gonna have to adapt yourself to production environment and work like a true professional. That means you gotta tolerate faulty situations and ambiguous specifications. (Quite often, you don't know WTF they want, but you'll eventually come around and hand in a piece of code that runs smoothly with the right output.) It's something you deal with every day if you work in a tech firm as an SDE, RSDE, etc..

In terms of this kind of blank-filling programming assignments, it's just fine for me. Some argue that they should allow us to write everything from scratch and judge the grades by the output, allowing more flexibility for the developemt process. I think the skeleton code is simple enough, leaving enough freedom for you to mess around. If you allow me to start from scratch, I'm gonna finish it with C++ or Python.

The one thing I say they should really fix is the instability of the grader. Random failures and insufferable waitings are the two things that take away most of the fun. The programming assignment itself wasn't so hard, but was made unnecessarily much harder by these two factors.

For the sake of the previous four courses, I can spare 4 stars. But I understand those 1 stars.

In my view this course fails on 3 fronts:

1) the construction of the exercises is quite poor: they don't build well on each other and they don't really produce a sensible application, because many optimization opportunities are ignored - for example, to avoid having to calculate the same thing multiple times, to ensure that lists are ordered for quicker searching etc. If the idea is to teach novices good application development this course fails miserably.

2) the fact is you can pass this course without writing a single unit test, and without producing a working application - in fact, you can get 10/10. This seems wrong for a 'capstone' project.

3) Because of the design of the exercises (the function signatures) it is hard to make use of Spark in a sensible way. Spark is surely a major reason many people follow the overall program. Why does this project not build on the Spark course? That would make it really interesting.

It's very well put together. It just requires a great deal of time dedication specially because blocks of time should be meaningul to not lose the context present in one's mind.

Outstanding set of courses.

Just to counter comments about grader stability and lack of grader feedback - I never encountered this in any of the 5 courses.

The topic of capstone is great, but the milestones, and their APIs direct towards bad solutions.

Milestone annoyingggggggggg

Buggy grader

I didn't learn anything new about Scala or Big Data from this project. It was more like how to "tune" the code so that I can pass the assignments. If you want to learn about maps and coordinates and all the math that goes with it, this is for you. Otherwise you're losing your time.

Im not finished yet, but its really nice one. Much place to play with data and optimization beyond actual course requirements. Lot of freedom in implementation wich is great. Ive already learned a lot and planning to spend time playnig with Observatory project further just for fun and experience. Thank you so much for this opportunity! :)

Excellent way of ending this specialization. This course covered almost everything was taught during the other 4 courses and gave a concrete real world usage of it. It helps to see the knowledge in action in a very cool way. Definitely take this course, the feeling of satisfaction at the end alone makes it worth the effort.

Incredibly well put together. The capstone project is a great exercise in assimilating all the knowledge from the other related courses in function programming package. I like that I had to learn about interesting algorithms like the Mercator projection and inverse-distance weighting.

The perfect end of the course. It manages to recap all the concept, the libraries and the exercises that has been done during all other courses. It is a good point to remember all the things that one has been made, and a good point to review all the things that were not so clear.

Phenominal course! Well organized, good pace, great combination of "practical theory" with examples that illustrate the concepts. and very clear well-thought out explanations by Dr. Miller. Best Spark class I've seen, and best class in this series (sorry, Dr. Odersky).

Awesome project!. It is not only wrapping all the previous concepts in a real use case, but also introducing a lot of different algorithms! Calculate deviations, spatial interpolation, great-circle distance, inverse distance weighting, etc.

Most of the case, it is hard to deal with big data. This course provides an opportunity to handle big data and visualize the handled result on UI. Also, as you learned from FPD in Scala course, UI is developed with Scala.js

Kind of a long homework, quite open-ended in the sense that implementations are not guided step by step by the instructor. Some of the tasks can be approached in different ways and can be quite challenging at time.

The capstone project has done a fantastic job of drilling in some of the important fundamentals taught in the rest of the courses. It is definitely worth taking if you've done the rest of the courses.

An interesting project that nicely covers most concepts discussed in the previous 4 Scala/Spark courses. The provided code skeleton is very helpful.