-
Metrics integration for OkHttp. looks quite nice
How Completely Messed Up Practices Become Normal
on Normalization of Deviance, with a few anecdotes from Silicon Valley. "The gradual process through which unacceptable practice or standards become acceptable. As the deviant behavior is repeated without catastrophic results, it becomes the social norm for the organization."
normalization-of-deviance deviance bugs culture ops reliability work workplaces processes norms
2015
A critical analysis of the Legacy Verified SSL/TLS proposal by CloudFlare & Facebook
The history of real-world CA-based PKI is pretty awful
Incredibly Rare Underwater Footage of a Stray Giant Squid Swimming Around Toyama Bay in Japan
wow, this is great footage
-
hooray, Docker registry here at last
How to inspect SSL/TLS traffic with Wireshark 2
turns out it's easy enough -- Mozilla standardised a debugging SSL session-key logging file format which Wireshark and Chrome support
ImperialViolet - Juniper: recording some Twitter conversations
Adam Langley on the Juniper VPN-snooping security hole:
... if it wasn't the NSA who did this, we have a case where a US government backdoor effort (Dual-EC) laid the groundwork for someone else to attack US interests. Certainly this attack would be a lot easier given the presence of a backdoor-friendly RNG already in place. And I've not even discussed the SSH backdoor. [...]
primes ecc security juniper holes exploits dual-ec-drbg vpn networking crypto prngs
Excellent post from Matthew Green on the Juniper backdoor
For the past several years, it appears that Juniper NetScreen devices have incorporated a potentially backdoored random number generator, based on the NSA's Dual_EC_DRBG algorithm. At some point in 2012, the NetScreen code was further subverted by some unknown party, so that the very same backdoor could be used to eavesdrop on NetScreen connections. While this alteration was not authorized by Juniper, it's important to note that the attacker made no major code changes to the encryption mechanism -- they only changed parameters. This means that the systems were potentially vulnerable to other parties, even beforehand. Worse, the nature of this vulnerability is particularly insidious and generally messed up. [....] The end result was a period in which someone -- maybe a foreign government -- was able to decrypt Juniper traffic in the U.S. and around the world. And all because Juniper had already paved the road. One of the most serious concerns we raise during [anti-law-enforcement-backdoor] meetings is the possibility that encryption backdoors could be subverted. Specifically, that a back door intended for law enforcement could somehow become a backdoor for people who we don't trust to read our messages. Normally when we talk about this, we're concerned about failures in storage of things like escrow keys. What this Juniper vulnerability illustrates is that the danger is much broader and more serious than that. The problem with cryptographic backdoors is not that they're the only way that an attacker can break intro our cryptographic systems. It's merely that they're one of the best. They take care of the hard work, the laying of plumbing and electrical wiring, so attackers can simply walk in and change the drapes.
(via Tony Finch)via:fanf crypto backdoors politics juniper dual-ec-drbg netscreen vpn
-
good thread of AWS' shortcomings -- so many services still don't handle VPC for instance
Big Brother is born. And we find out 15 years too late to stop him - The Register
During the passage of RIPA, and in many debates since 2000, Parliament was asked to consider and require data retention by telephone companies, claiming that the information was vital to fighting crime and terrorism. But Prime Minister Tony Blair and successive Home Secretaries David Blunkett and Jack Straw never revealed to Parliament that at the same time, the government was constantly siphoning up and storing all telephone call records at NTAC. As a result, MPs and peers spent months arguing about a pretence, and in ignorance of the cost and human rights implications of what successive governments were doing in secret.
ripa big-brother surveillance preston uk gchq mi5 law snooping
How to host Hugo static website generator on AWS Lambda
seriously, AWS. editing JSON files in a browser text box is an awful, awful user experience
-
A German bank offering a worldwide(?) bank account, using your smartphone (with push notifications etc.) as the main UI
The mystery of the power bank phone taking over Ghana
tl;dr: it's being used as a cheap, portable power bank
Gardai find 70 stolen bikes in one house being readied for export
The Limerick Leader quoted other unnamed gardai who said they believed those who had stolen the bikes were selling them to a third party for shipment abroad, most likely to another country in Europe. "It would seem that he has his own network on the Continent and has a lucrative market for the bikes he sends on," said one of the sources quoted in the report. "Some of the racing bikes would fetch large sums of money on the Continent." Trucks were seen arriving and departing the house in Castletroy where the find was made. And while it was unclear exactly how gardai were informed of the suspicious activity, when a team of officers went to search the property they found the bikes in the back garden.
-
"Statistical regression to the mean predicts that patients selected for abnormalcy will, on the average, tend to improve. We argue that most improvements attributed to the placebo effect are actually instances of statistical regression."
medicine science statistics placebo evidence via:hn regression-to-the-mean
League of Legends win-rates vs latency analysed
It appears that more mechanically intensive champions are more affected by latency, while tankier champions or those with point-and-click abilities are less affected by latency.
(via Nelson)games league-of-legends latency ping gaming internet via:nelson
-
via Tony Finch. 'In this post I will demonstrate how to do reservoir sampling orders of magnitude faster than the traditional "naive" reservoir sampling algorithm, using a fast high-fidelity approximation to the reservoir sampling-gap distribution.'
statistics reservoir-sampling sampling algorithms poisson bernoulli performance
The Moral Failure of Computer Scientists - The Atlantic
Phillip Rogaway, a professor of CS at UC Davis, contends that computer scientists should stand up against the construction of surveillance states built using their work:
Waddell: In your paper, you compare the debate over nuclear science in the 1950s to the current debate over cryptography. Nuclear weapons are one of the most obvious threats to humanity today — do you think surveillance presents a similar type of danger? Rogaway: I do. It's of a different nature, obviously. The threat is more indirect and more subtle. So with nuclear warfare, there was this visually compelling and frightening risk of going up in a mushroom cloud. And with the transition to a state of total surveillance, what we have is just the slow forfeiture of democracy.
ethics cryptography crypto surveillance politics phillip-rogaway morals speaking-out government
-
This is basically terrifying. A catalog of race conditions and reliability horrors around the POSIX filesystem abstraction in Linux -- it's a wonder anything works. 'Where's this documented? Oh, in some mailing list post 6-8 years ago (which makes it 12-14 years from today). The fs devs whose posts I've read are quite polite compared to LKML's reputation, and they generously spend a lot of time responding to basic questions, but it's hard for outsiders to troll [sic] through a decade and a half of mailing list postings to figure out which ones are still valid and which ones have been obsoleted! I don't mean to pick on filesystem devs. In their OSDI 2014 talk, the authors of the paper we're discussing noted that when they reported bugs they'd found, developers would often respond "POSIX doesn't let filesystems do that", without being able to point to any specific POSIX documentation to support their statement. If you've followed Kyle Kingsbury's Jepsen work, this may sound familiar, except devs respond with "filesystems don't do that" instead of "networks don't do that".I think this is understandable, given how much misinformation is out there. Not being a filesystem dev myself, I'd be a bit surprised if I don't have at least one bug in this post.'
filesystems linux unix files operating-systems posix fsync osdi papers reliability
[LUCENE-6917] Deprecate and rename NumericField/RangeQuery to LegacyNumeric - ASF JIRA
Interesting performance-related tweak going into Lucene -- based on the Bkd-Tree I think: https://users.cs.duke.edu/~pankaj/publications/papers/bkd-sstd.pdf . Being used for all numeric index types, not just multidimensional ones?
lucene performance algorithms patches bkd-trees geodata numeric indexing
Kevin Lyda's mega pension post
Cutting and pasting from Facebook for posterity... there are some really solid tips in here. 'Some people plan their lives out and then there are people like me who randomly do things and suddenly, in retrospect, it looks like a grand plan has come together. In reality it's more like my subconscious pulls in useful info and pokes me to go learn things as required. If you live/work in Ireland, the following "grand plan" might be useful. This year has apparently been "figure out how to retire" year. It started late last year with finally organising all my private Irish pensions (2 from employers, 1 personal). In the process I learned the following: * Many Irish pension plans allow you to start drawing down from them at age 50. There are downsides to this, but if you have several of them it allows you more room to avoid stock market downturns when you purchase annuities. * You can get 25% of each pension as a tax-free lump sum. I also learned a few property things. The key thing is that if you have a buy-to-let property you should *not* pay off its mortgage early. You can deduct 75% of the interest you pay against the taxes you'd owe for rental income. That means the interest you pay will essentially be close to or even under the rate of inflation. A residential mortgage might have a lower interest rate nominally, but the effective interest rate is higher. The Irish state pension is changing. If you are 68 after 2020 the rules have changed - and they're now much simpler. Work for 10 years and you get the minimum state pension (1/3 of a full pension). Work for 20, you get 2/3 of of a state pension. Work for 30, you get a full pension. But you can't collect it till you're 68 and remember that Irish employers can apparently force you to "retire" at 65 (ageism is legal). So you need to bridge those 3 years (or hope they change the law to stop employers from doing that). When I "retired" I kept a part time job for a number of reasons, but one was because I suspected I needed more PRSI credits for a pension. And it turns out this was correct. Part-time work counts as long as you make more than €38/week. And self-employment counts as long as you make more than €5,000/year. You can also make voluntary PRSI contributions (around €500/year but very situation dependent). If you've worked in Europe or the US or Canada or a few other countries, you can get credits for social welfare payments in those countries. But if you have enough here and you have enough for some pension in the other country, you can draw a pension from both. Lastly most people I've talked to about retirement this year have used the analogy of legs on a stool. Every source of post-retirement income is a leg on the stool - the more legs, the more secure your retirement. There are lots of options for legs: * Rental income. This is a little wobbly as legs go at least for me. But if you have more than one rental property - and better yet some commercial rental property - this leg firms up a bit. Still, it's a bit more work than most. * Savings. This isn't very tax-efficient, but it can help fill in blank spots some legs have (like rental income or age restrictions) or maximise another legs value (weathering downturns for stock-based legs). And in retirement you can even build savings up. Sell a house, the private pension lump sum, etc. But remember you're retired, go have fun. Savings won't do you much good when you're dead. * Stocks. I've cashed all mine in, but some friends have been more restrained in cashing in stocks they might have gotten from employers. This is a volatile leg, but it can pay off rather well if you know what you're doing. But be honest with yourself. I know I absolutely don't know what I'm doing on this so stayed away. * Government pension. This is generally a reliable source of income in retirement. It's usually not a lot, but it does tend to last from retirement to death and it shows up every month. You apply once and then it just shows up each month. If you've worked in multiple countries, you can hedge some bets by taking a pension in each country you qualify from. You did pay into them after all. * Private pension. This can also give you a solid source of income but you need to pay into it. And paying in during your 20s and 30s really pays off later. But you need to make your investments less risky as you get into your late 50s - so make sure to start looking at them then. And you need to provide yourself some flexibility for starting to draw it down in order to survive market drops. The crash in 2007 didn't fully recover until 2012 - that's 5 years. * Your home. Pay off your mortgage and your home can be a leg. Not having to pay rent/mortgage is a large expense removed and makes the other legs more effective. You can also "sell down" or look into things like reverse mortgages, but the former can take time and has costs while the latter usually seems to have a lot of fine print you should read up on. * Part-time work. I know a number of people who took part-time jobs when they retired. If you can find something that doesn't take a huge amount of time that you'd enjoy doing and that people will pay you for, fantastic! Do that. And it gets you out of the house and keeping active. For friends who are geeks and in my age cohort, I note that it will be 2037 around the time we hit 65. If you know why that matters, ka-ching!' Another particularly useful page about the state pension: "Six things every woman needs to know about the State pension", Irish Times, Dec 1 2015, https://www.irishtimes.com/business/personal-finance/six-things-every-woman-needs-to-know-about-the-state-pension-1.2448981 , which links to this page to get your state pension contribution record: http://www.welfare.ie/en/pages/secure/ RequestSIContributionRecord.aspx
pensions money life via:klyda stocks savings shares property ireland old-age retirement
-
As Glynn Moody noted, if UK police, intelligence agencies, HMRC and others call all legally hack phones and computers, that also means that digital evidence can be easily and invisibly planted. This will undermine future court cases in the UK, which seems like a significant own goal...
hmrc police gchq uk hacking security law-enforcement evidence law
Why We Chose Kubernetes Over ECS
3 months ago when we, at nanit.com, came to evaluate which Docker orchestration framework to use, we gave ECS the first priority. We were already familiar with AWS services, and since we already had our whole infrastructure there, it was the default choice. After testing the service for a while we had the feeling it was not mature enough and missing some key features we needed (more on that later), so we went to test another orchestration framework: Kubernetes. We were glad to discover that Kubernetes is far more comprehensive and had almost all the features we required. For us, Kubernetes won ECS on ECS's home court, which is AWS.
Beachbum Berry — Latitude 29 Formula Orgeat
The legendary Jeff "Beachbum" Berry, tiki-cocktail wizard, has partnered with a Brooklyn-based orgeat maker to provide the key ingredient for an original Trader-Vic-style Mai Tai. may be a bit tricky to ship to Ireland though!
How to Spot Bitcoin Inventor Satoshi Nakamoto | MIT Technology Review
Emin Gün Sirer pours cold water on the "Craig Wright is Satoshi Nakamoto" theory
Dr TJ McIntyre: Fight against cybercrime needs funding, not more words - Independent.ie
Is the Irish policing system capable of tackling computer crime? A report this week from the Garda Inspectorate makes it clear that the answer is no. There is no Garda cybercrime unit, which is of serious concern given the threat posed by cybercrime to key national infrastructure such as energy, transport and telecommunications systems. [...] A combination of inadequate resources and increased workload have swamped the [Computer Crime Investigation Unit]. Today, almost every crime is a computer crime, in the sense that mobile phones, laptops and even devices such as game consoles are likely to contain evidence. The need to forensically inspect all these devices - using outdated equipment - has resulted in several-year delays and seem to have forced the unit into a position where it is running to stand still rather than responding to new developments.
-
I keep having to google this, so here's a good one which works -- unlike Wolfram Alpha!
birthday birthday-paradox birthday-problem hashes hash-collision attacks security collisions calculators probability statistcs
-
'At least for Europe it is obvious: All roads lead to Rome! You can reach the eternal city on almost 500.000 routes from all across the continent. Which road would you take? To approach one of the biggest unsolved quests of mobility, the first question we asked ourselves was: Where do you start, when you want to know every road to Rome? We aligned starting points in a 26.503.452 km² grid covering all of Europe. Every cell of this grid contains the starting point to one of our journeys to Rome. Now that we have our 486.713 starting points we need to find out how we could reach Rome as our destination. For this we created a algorithm that calculates one route for every trip. The more often a single street segment is used, the stronger it is drawn on the map. The maps as outcome of this project is somewhere between information visualization and data art, unveiling mobility and a very large scale.' Beautiful! Decent-sized prints available for 26 euros too.
Tools for debugging, testing and using HTTP/2
excellent, extensive list from Cloudflare
AWS Api Gateway for Fun and Profit
good worked-through example of an API Gateway rewriting system
api-gateway aws api http services ops alerting alarming opsgenie signalfx
EU counter-terror bill is 'indiscriminate' data sweep
"To identify if someone is travelling outside the EU, we don't need an EU PNR. This data are already easily available in the airline reservation system," [Giovanni Buttarelli, the European data protection supervisor] said. EU governments want more information in the belief it will help law enforcement in tracking down terrorists and are demanding access to information, such as travel dates, travel itinerary, ticket information, contact details, baggage information, and payment information of anyone flying in or out of the EU. ... EU PNR data would be retained for up to five years
pnr eu law privacy data-protection europe counter-terrorism travel air-travel
Fast Forward Labs: Fashion Goes Deep: Data Science at Lyst
this is more than just data science really -- this is proper machine learning, with deep learning and a convolutional neural network. serious business
lyst machine-learning data-science ml neural-networks supervised-learning unsupervised-learning deep-learning
Why Percentiles Don't Work the Way you Think
Baron Schwartz on metrics, percentiles, and aggregation. +1, although as a HN commenter noted, quantile digests are probably the better fix
performance percentiles quantiles statistics metrics monitoring baron-schwartz vividcortex
-
Spotify wrote their own metrics store on ElasticSearch and Cassandra. Sounds very similar to Prometheus
ELS: latency based load balancer, part 1
ELS measures the following things: Success latency and success rate of each machine; Number of outstanding requests between the load balancer and each machine. These are the requests that have been sent out but we haven't yet received a reply; Fast failures are better than slow failures, so we also measure failure latency for each machine. Since users care a lot about latency, we prefer machines that are expected to answer quicker. ELS therefore converts all the measured metrics into expected latency from the client's perspective.[...] In short, the formula ensures that slower machines get less traffic and failing machines get much less traffic. Slower and failing machines still get some traffic, because we need to be able to detect when they come back up again.
latency spotify proxies load-balancing els algorithms c3 round-robin load-balancers routing
Low-latency journalling file write latency on Linux
great research from LMAX: xfs/ext4 are the best choices, and they explain why in detail, referring to the code
linux xfs ext3 ext4 filesystems lmax performance latency journalling ops
-
nice 3D printed maps from this Irish company
-
"Irish police have no cybercrime unit, and 1/3 of police have no email." ffs!
cybercrime policing ireland gardai fraud privacy phishing hacking internet law
A Gulp Workflow for Amazon Lambda
'any nontrivial development of Lambda functions will require a simple, automated build/deploy process that also fills a couple of Lambda's gaps such as the use of node modules and environment variables.' See also https://medium.com/@AdamRNeary/developing-and-testing-amazon-lambda-functions-e590fac85df4#.mz0a4qk3j : 'I am psyched about Amazon's new Lambda service for asynchronous task processing, but the ideal development and testing cycle is really left to the engineer. While Amazon provides a web-based console, I prefer an approach that uses Mocha. Below you will find the gritty details using Kinesis events as a sample input.'
lambda aws services testing deployment ops mocha gulp javascript
"Hidden Technical Debt in Machine-Learning Systems" [pdf]
Another great paper about from Google, talking about the tradeoffs that must be considered in practice over the long term with running a complex ML system in production.
technical-debt ml machine-learning ops software production papers pdf google
Introducing Netty-HTTP from Cask
netty-http library solves [Netty usability issues] by using JAX-RS annotations to build a HTTP path routing layer on top of netty. In addition, the library implements a guava service to manage the HTTP service. netty-http allows users of the library to just focus on writing the business logic in HTTP handlers without having to worry about the complexities of path routing or learning netty pipeline internals to build the HTTP service.
We've written something very similar, although I didn't even bother supporting JAX-RS annotations -- just a simple code-level DSL.The Locals Xmas Gift Guide 2015
some nice local gift suggestions from small businesses around Dublin. I'd love to get some of these, but I guess I'll have to settle for giving them instead ;)
Topics in High-Performance Messaging
'We have worked together in the field of high-performance messaging for many years, and in that time, have seen some messaging systems that worked well and some that didn't. Successful deployment of a messaging system requires background information that is not easily available; most of what we know, we had to learn in the school of hard knocks. To save others a knock or two, we have collected here the essential background information and commentary on some of the issues involved in successful deployments. This information is organized as a series of topics around which there seems to be confusion or uncertainty. Please contact us if you have questions or comments.'
messaging scalability scaling performance udp tcp protocols multicast latency
Intercom Engineering Insights - Scale and Reliability 2015
next Intercom hiring^Wevent coming up, Dec 10th in Dublin, talking about how they scale and ops their ElasticSearch and Mongo clusters
Control theory meets machine learning
'DB: Is there a difference between how control theorists and machine learning researchers think about robustness and error? BR: In machine learning, we almost always model our errors as being random rather than worst-case. In some sense, random errors are actually much more benign than worst-case errors. [...] In machine learning, by assuming average-case performance, rather than worst-case, we can design predictive algorithms by averaging out the errors over large data sets. We want to be robust to fluctuations in the data, but only on average. This is much less restrictive than the worst-case restrictions in controls. DB: So control theory is model-based and concerned with worst case. Machine learning is data based and concerned with average case. Is there a middle ground? BR: I think there is! And I think there's an exciting opportunity here to understand how to combine robust control and reinforcement learning. Being able to build systems from data alone simplifies the engineering process, and has had several recent promising results. Guaranteeing that these systems won't behave catastrophically will enable us to actually deploy machine learning systems in a variety of applications with major impacts on our lives. It might enable safe autonomous vehicles that can navigate complex terrains. Or could assist us in diagnostics and treatments in health care. There are a lot of exciting possibilities, and that's why I'm excited about how to find a bridge between these two viewpoints.'
control-theory interviews machine-learning ml worst-case self-driving-cars cs
-
This is my bet: the age of dynamic languages is over. There will be no new successful ones. Indeed we have learned a lot from them. We've learned that library code should be extendable by the programmer (mixins and meta-programming), that we want to control the structure (macros), that we disdain verbosity. And above all, we've learned that we want our languages to be enjoyable. But it's time to move on. We will see a flourishing of languages that feel like you're writing in a Clojure, but typed. Included will be a suite of powerful tools that we've never seen before, tools so convincing that only ascetics will ignore.
programming scala clojure coding types strong-types dynamic-languages languages
-
'IRC without netsplits' using Raft consensus
Inside China's Memefacturing Factories, Where The Hottest New Gadgets Are Made - BuzzFeed News
On a humid afternoon, Zhou went shopping for some of those very parts at a Bao An market. As he pulled his maroon minivan into a crowded parking lot, the full scale of Depu Electronics came into view: a three-story concrete behemoth roughly bigger than a Costco and roughly smaller than the Pentagon. Inside, it looked like the world's largest Radio Shack going out of business sale: an endless series of booths with cables and circuit boards and plugs and ports and buttons and machines piled so high on tables that the faces of the clerks who were selling them were hidden from view. Each booth seemed to argue: We have exactly what you want and we have enough of it for all of your customers. Short of motorized wheels and molding, the market offered nearly everything an ambitious factory owner would need to build a hoverboard, just waiting to be bought, assembled, and shipped.
hoverboards memes china manufacturing future gadgets tat bao-an electronics
One of the Largest Hacks Yet Exposes Data on Hundreds of Thousands of Kids | Motherboard
VTech got hacked, and millions of parents and 200,000 kids had their privacy breached as a result. Bottom line is summed up by this quote from one affected parent:
"Why do you need know my address, why do you need to know all this information just so I can download a couple of free books for my kid on this silly pad thing? Why did they have all this information?"
Quite. Better off simply not to have the data in the first place!Senior Anglo bondholders revealed in department note
In case you were wondering who Ireland's economy was wiped out for:
Among the major holders were a Dutch pension fund, ABP; another Dutch fund, PGGM; LGPI in Finland, which manages local government pensions; and a Swiss public entities pension. A number of major asset managers were also named, including JP Morgan in London; DeKA and ADIG, two German investment managers; and Robeco from the Netherlands. Big insurance companies, including Munich Re, Llmarinen from Finland and German giant Axa were also named, along with big banks such as BNP, SocGen, ING and Deutsche.
bondholders anglo economy ireland politics eu senior-bondholders
-
a bunch of metrics for Dublin xmas-shopping capacity
re:Work - The five keys to a successful Google team
We learned that there are five key dynamics that set successful teams apart from other teams at Google: Psychological safety: Can we take risks on this team without feeling insecure or embarrassed? Dependability: Can we count on each other to do high quality work on time? Structure & clarity: Are goals, roles, and execution plans on our team clear? Meaning of work: Are we working on something that is personally important for each of us? Impact of work: Do we fundamentally believe that the work we're doing matters?
-
75%. This is really quite tricky!
Art Meets Cartography: The 15,000-Year History of a River in Oregon Rendered in Data
this is really beautiful. Available as a printable, 17" x 38" PDF from http://www.oregongeology.org/pubs/ll/p-poster-willamette.htm
art data mapping geodata oregon rivers willamette-river history lidar
Accretion Disc Series - Clint Fulkerson
available as prints -- vector art with a hint of the bacterial
algorithms art graphics vector bacteria petri-dish clint-fulkerson
John Nagle on delayed ACKs and his algorithm
love it when things like this show up
-
She thought they were a normal couple until she found a passport in a glovebox – and then her world shattered. Now she is finally getting compensation and a police apology for that surreal, state-sponsored deception. But she still lies awake and wonders: did he ever really love me?
I can't believe this was going on in the 2000s!surveillance police uk undercover scandals policing environmentalism greens
Just use /dev/urandom to generate random numbers
Using SHA-1 [to generate random numbers] in this way, with a random seed and a counter, is just building a (perfectly sound) CSPRNG with, I believe, an 80-bit security level. If you trust the source of the random seed, e.g. /dev/urandom, you may as well just use /dev/urandom itself. If you don't, you're already in trouble. And if you somehow need a userspace PRNG, the usual advice about not rolling your own crypto unless you know what you're doing applies. (Especially for database IDs, the risk of collisions should be considered a security problem, ergo this should be considered crypto, until proven otherwise.) In this case, using BLAKE2 instead of SHA-1 would get you a higher security level and faster hashing. Or, in tptacek's words: http://sockpuppet.org/blog/2014/02/25/safely-generate-random-numbers/
Authenticated app packages on Sandstorm with PGP and Keybase
Nice approach to package authentication UX using Keybase/PGP.
When you go to install a package, Sandstorm verifies that the package is correctly signed by the Ed25519 key. It looks for a PGP signature in the metadata, and verifies that the PGP-signed assertion is for the correct app ID and the email address specified in the metadata. It queries the Keybase API to see what accounts the packager has proven ownership of, and lists them with their links on the app install page.
authentication auth packages sandstorm keybase pgp gpg security
-
Floating car data (FCD), also known as floating cellular data, is a method to determine the traffic speed on the road network. It is based on the collection of localization data, speed, direction of travel and time information from mobile phones in vehicles that are being driven. These data are the essential source for traffic information and for most intelligent transportation systems (ITS). This means that every vehicle with an active mobile phone acts as a sensor for the road network. Based on these data, traffic congestion can be identified, travel times can be calculated, and traffic reports can be rapidly generated. In contrast to traffic cameras, number plate recognition systems, and induction loops embedded in the roadway, no additional hardware on the road network is necessary.
surveillance cars driving mobile-phones phones travel gsm monitoring anpr alpr traffic
CiteSeerX — The Confounding Effect of Class Size on the Validity of Object-oriented Metrics
A lovely cite from @conor. Turns out the sheer size of an OO class is itself a solid fault-proneness metric
metrics coding static-analysis error-detection faults via:conor oo
How a group of neighbors created their own Internet service | Ars Technica
Orcas Island, WA. impressive stuff
Report: Everyone Should Get a Security Freeze
"Whether your personal information has been stolen or not, your best protection against someone opening new credit accounts in your name is the security freeze (also known as the credit freeze), not the often-offered, under-achieving credit monitoring. Paid credit monitoring services in particular are not necessary because federal law requires each of the three major credit bureaus to provide a free credit report every year to all customers who request one. You can use those free reports as a form of do-it-yourself credit monitoring."
Even the LastPass Will be Stolen, Deal with It
ugh, quite a long list of LastPass security issues
Signs Point to Unencrypted Communications Between Terror Suspects
News emerging from Paris — as well as evidence from a Belgian ISIS raid in January — suggests that the ISIS terror networks involved were communicating in the clear, and that the data on their smartphones was not encrypted.
Global Continuous Delivery with Spinnaker
Netflix' CD platform, post-Atlas. looks interesting
continuous-delivery aws netflix cd devops ops atlas spinnaker
-
by reordering items to optimize locality. Via aphyr's dad!
caches cache-friendly optimization data-locality performance coding algorithms
Reporting Error Leads To Speculation That Terrorists Used PS4s To Plan Paris Attacks
lol. Nice work, Forbes
forbes fail ps4 crypto terrorism reporting msm speculation hysteria
Did you know that Dublin Airport is recording your phone's data? - Newstalk
Ugh. Queue tracking using secret MAC address tracking in Dublin Airport:
"I think the fundamental issue is one of consent. Dublin Airport have been tracking individual MAC addresses since 2012 and there doesn't appear to be anywhere in the airport where they warn passengers that this is this occurring. "If they have to signpost CCTV, then mobile phone tracking should at a very minimum be sign-posted for passengers," he continues.
And how long are MAC addresses retained for, I wonder?mac-addresses dublin-airport travel privacy surveillance tracking wifi phones cctv consent
Pinboard on the Next Economy Conference (with tweets)
Maciej Ceglowski went to an O'Reilly SV-boosterish conference and produced these excellent tweets
twitter conferences oreilly silicon-valley new-economy future lyft uber unions maciej-ceglowski
Our Generation Ships Will Sink / Boing Boing
Kim Stanley Robinson on the feasibility of interstellar colonization: 'There is no Planet B! Earth is our only possible home!'
The impact of Docker containers on the performance of genomic pipelines [PeerJ]
In this paper, we have assessed the impact of Docker containers technology on the performance of genomic pipelines, showing that container "virtualization" has a negligible overhead on pipeline performance when it is composed of medium/long running tasks, which is the most common scenario in computational genomic pipelines. Interestingly for these tasks the observed standard deviation is smaller when running with Docker. This suggests that the execution with containers is more "homogeneous," presumably due to the isolation provided by the container environment. The performance degradation is more significant for pipelines where most of the tasks have a fine or very fine granularity (a few seconds or milliseconds). In this case, the container instantiation time, though small, cannot be ignored and produces a perceptible loss of performance.
Three quarters of cars stolen in France 'electronically hacked' - Telegraph
The astonishing figures come two months after computer scientists in the UK warned that thousands of cars – including high-end brands such as Porsches and Maseratis - are at risk of electronic hacking. Their research was suppressed for two years by a court injunction for fear it would help thieves steal vehicles to order. The kit required to carry out such "mouse jacking", as the French have coined the practice, can be freely purchased on the internet for around £700 and the theft of a range of models can be pulled off "within minutes," motor experts warn.
hacking security security-through-obscurity mouse-jacking cars safety theft crime france smart-cars
-
Awesome new mock DynamoDB implementation:
An implementation of Amazon's DynamoDB, focussed on correctness and performance, and built on LevelDB (well, @rvagg's awesome LevelUP to be precise). This project aims to match the live DynamoDB instances as closely as possible (and is tested against them in various regions), including all limits and error messages. Why not Amazon's DynamoDB Local? Because it's too buggy! And it differs too much from the live instances in a number of key areas.
We use DynamoDBLocal in our tests -- the availability of that tool is one of the key reasons we have adopted Dynamo so heavily, since we can safely test our code properly with it. This looks even better.dynamodb testing unit-tests integration-testing tests ops dynalite aws leveldb
Alarm design: From nuclear power to WebOps
Imagine you are an operator in a nuclear power control room. An accident has started to unfold. During the first few minutes, more than 100 alarms go off, and there is no system for suppressing the unimportant signals so that you can concentrate on the significant alarms. Information is not presented clearly; for example, although the pressure and temperature within the reactor coolant system are shown, there is no direct indication that the combination of pressure and temperature mean that the cooling water is turning into steam. There are over 50 alarms lit in the control room, and the computer printer registering alarms is running more than 2 hours behind the events. This was the basic scenario facing the control room operators during the Three Mile Island (TMI) partial nuclear meltdown in 1979. The Report of the President's Commission stated that, "Overall, little attention had been paid to the interaction between human beings and machines under the rapidly changing and confusing circumstances of an accident" (p. 11). The TMI control room operator on the day, Craig Faust, recalled for the Commission his reaction to the incessant alarms: "I would have liked to have thrown away the alarm panel. It wasn't giving us any useful information". It was the first major illustration of the alarm problem, and the accident triggered a flurry of human factors/ergonomics (HF/E) activity.
A familiar topic for this ex-member of the Amazon network monitoring team...ergonomics human-factors ui ux alarms alerts alerting three-mile-island nuclear-power safety outages ops
An Analysis of Reshipping Mule Scams
We observed that the vast majority of the re-shipped packages end up in the Moscow, Russia area, and that the goods purchased with stolen credit cards span multiple categories, from expensive electronics such as Apple products, to designer clothes, to DSLR cameras and even weapon accessories. Given the amount of goods shipped by the reshipping mule sites that we analysed, the annual revenue generated from such operations can span between 1.8 and 7.3 million US dollars. The overall losses are much higher though: the online merchant loses an expensive item from its inventory and typically has to refund the owner of the stolen credit card. In addition, the rogue goods typically travel labeled as "second hand goods" and therefore custom taxes are also evaded. Once the items purchased with stolen credit cards reach their destination they will be sold on the black market by cybercriminals. [...] When applying for the job, people are usually required to send the operator copies of their ID cards and passport. After they are hired, mules are promised to be paid at the end of their first month of employment. However, from our data it is clear that mules are usually never paid. After their first month expires, they are never contacted back by the operator, who just moves on and hires new mules. In other words, the mules become victims of this scam themselves, by never seeing a penny. Moreover, because they sent copies of their documents to the criminals, mules can potentially become victims of identity theft.
crime law cybercrime mules shipping-scams identity-theft russia moscow scams papers
No Harm, No Fowl: Chicken Farm Inappropriate Choice for Data Disposal
That's a lesson that Spruce Manor Special Care Home in Saskatchewan had to learn the hard way (as surprising as that might sound). As a trustee with custody of personal health information, Spruce Manor was required under section 17(2) of the Saskatchewan Health Information Protection Act to dispose of its patient records in a way that protected patient privacy. So, when Spruce Manor chose a chicken farm for the job, it found itself the subject of an investigation by the Saskatchewan Information and Privacy Commissioner. In what is probably one of the least surprising findings ever, the commissioner wrote in his final report that "I recommend that Spruce Manor […] no longer use [a] chicken farm to destroy records", and then for good measure added "I find using a chicken farm to destroy records unacceptable."
data law privacy funny chickens farming via:pinboard data-protection health medical-records
Caffeine cache adopts Window TinyLfu eviction policy
'Caffeine is a Java 8 rewrite of Guava's cache. In this version we focused on improving the hit rate by evaluating alternatives to the classic least-recenty-used (LRU) eviction policy. In collaboration with researchers at Israel's Technion, we developed a new algorithm that matches or exceeds the hit rate of the best alternatives (ARC, LIRS). A paper of our work is being prepared for publication.' Specifically:
W-TinyLfu uses a small admission LRU that evicts to a large Segmented LRU if accepted by the TinyLfu admission policy. TinyLfu relies on a frequency sketch to probabilistically estimate the historic usage of an entry. The window allows the policy to have a high hit rate when entries exhibit a high temporal / low frequency access pattern which would otherwise be rejected. The configuration enables the cache to estimate the frequency and recency of an entry with low overhead. This implementation uses a 4-bit CountMinSketch, growing at 8 bytes per cache entry to be accurate. Unlike ARC and LIRS, this policy does not retain non-resident keys.
tinylfu caches caching cache-eviction java8 guava caffeine lru count-min sketching algorithms
-
The ever-shitty Java serialization creates a security hole
-
Danish glassware artist making wonderful Wunderkammers -- cabinets of curiosities --- entirely from glass. Seeing as one of his works sold for UKP50,000 last year, I suspect these are a bit out of my league, sadly
London garden bridge users to have mobile phone signals tracked
If it goes ahead, people's progress across the structure would be tracked by monitors detecting the Wi-Fi signals from their phones, which show up the device's Mac address, or unique identifying code. The Garden Bridge Trust says it will not store any of this data and is only tracking phones to count numbers and prevent overcrowding.
Red lines and no-go zones - the coming surveillance debate
The Anderson Report to the House of Lords in the UK on RIPA introduces a concept of a "red line":
"Firm limits must also be written into the law: not merely safeguards, but red lines that may not be crossed." … "Some might find comfort in a world in which our every interaction and movement could be recorded, viewed in real time and indefinitely retained for possible future use by the authorities. Crime fighting, security, safety or public health justifications are never hard to find." [13.19] The Report then gives examples, such as a perpetual video feed from every room in every house, the police undertaking to view the record only on receipt of a complaint; blanket drone-based surveillance; licensed service providers, required as a condition of the licence to retain within the jurisdiction a complete plain-text version of every communication to be made available to the authorities on request; a constant data feed from vehicles, domestic appliances and health-monitoring personal devices; fitting of facial recognition software to every CCTV camera and the insertion of a location-tracking chip under every individual's skin. It goes on: "The impact of such powers on the innocent could be mitigated by the usual apparatus of safeguards, regulators and Codes of Practice. But a country constructed on such a basis would surely be intolerable to many of its inhabitants. A state that enjoyed all those powers would be truly totalitarian, even if the authorities had the best interests of its people at heart." [13.20] … "The crucial objection is that of principle. Such a society would have gone beyond Bentham's Panopticon (whose inmates did not know they were being watched) into a world where constant surveillance was a certainty and quiescence the inevitable result. There must surely come a point (though it comes at different places for different people) where the escalation of intrusive powers becomes too high a price to pay for a safer and more law abiding environment." [13.21]
panopticon jeremy-bentham law uk dripa ripa surveillance spying police drones facial-recognition future tracking cctv crime
Dublin is a medium-density city
Comparable to Copenhagen or Amsterdam, albeit without sufficient cycling/public-transport infrastructural investment
infrastructure density housing dublin ireland cities travel commuting cycling
-
I'm tired of this shit. Full stop tired. It's 2015 and these turds who grope their way around conferences and the like can make allegations like this, get a hand wave and an, "Oh, that's just crazy Raymond!" Fuck that. Fuck it from here to hell and back. Here's a man who really hasn't done anything all that special, is a totally crazy gun-toting misogynist of the highest order and, yet, he remains mostly unchallenged after the tempest dies down, time after time. [...] I'm sure ESR will still be haunting conferences when your daughters reach their professional years unless you get serious about outing the assholes like him and making the community a lot less toxic than it is now.?
Amen to that.
User data plundering by Android and iOS apps is as rampant as you suspected
An app from Drugs.com, meanwhile, sent the medical search terms "herpes" and "interferon" to five domains, including doubleclick.net, googlesyndication.com, intellitxt.com, quantserve.com, and scorecardresearch.com, although those domains didn't receive other personal information.
Volkswagen emissions cheating was technical debt
Is this the first case of tech debt costing $18 billion?
"Perhaps the engineers told themselves that the cheat was a stopgap, and they'd address it later. If so, they didn't."
Nobody Loves Graphite Anymore - VividCortex
Graphite has a place in our current monitoring stack, and together with StatsD will always have a special place in the hearts of DevOps practitioners everywhere, but it's not representative of state-of-the-art in the last few years. Graphite is where the puck was in 2010. If you're skating there, you're missing the benefits of modern monitoring infrastructure. The future I foresee is one where time series capabilities (the raw power needed, which I described in my time series requirements blog post, for example) are within everyone's reach. That will be considered table stakes, whereas now it's pretty revolutionary.
Like I've been saying -- we need Time Series As A Service! This should be undifferentiated heavy lifting.graphite tsd time-series vividcortex statsd ops monitoring metrics
-
PICO-8 is a fantasy console for making, sharing and playing tiny games and other computer programs. When you turn it on, the machine greets you with a shell for typing in Lua commands and provides simple built-in tools for creating your own cartridges.
So cute! See also Voxatron, something similar for voxel-oriented 3D gaming Why Static Website Generators Are The Next Big Thing
Now _this_ makes me feel old. Alternative title: "why static website generators have been a good idea since WebMake, 15 years ago". WebMake does pretty well on the checklist of "key features of the modern static website generator", which are: 1. Templating (check); 2. Markdown support (well, EtText, which predated Markdown by several years); 3. Metadata (check); and 4. Javascript asset pipeline (didn't support this one, since complex front-end DHTML JS wasn't really a thing at the turn of the century. But I would have if it had ;). So I guess I was on the right track!
web html history webmake static-sites bake-dont-fry site-generators cms
Food Trucks Are Great Incubators. Why Don't We Have More?
So is that kind of thriving food-truck scene something the city should work to encourage? Theresa Hernandez, one of the owners of K Chido Mexico, thinks so. "There's a whole market there for a new culture," she says. "There's no doubt about it, the appetite is there. It's just a matter for somebody who is innovative enough in Dublin City Council to say: 'Right, let's do this.'"
Amen to that.wangle/Codel.h at master · facebook/wangle
Facebook's open-source implementation of the CoDel queue management algorithm applied to server request-handling capacity in their C++ service bootstrap library, Wangle.
wangle facebook codel services capacity reliability queueing
-
Despite its overarching abstractions, it is semantically non-uniform and its complicated transaction and job scheduling heuristics ordered around a dependently networked object system create pathological failure cases with little debugging context that would otherwise not necessarily occur on systems with less layers of indirection. The use of bus APIs complicate communication with the service manager and lead to duplication of the object model for little gain. Further, the unit file options often carry implicit state or are not sufficiently expressive. There is an imbalance with regards to features of an eager service manager and that of a lazy loading service manager, having rusty edge cases of both with non-generic, manager-specific facilities. The approach to logging and the circularly dependent architecture seem to imply that lots of prior art has been ignored or understudied.
analysis systemd linux unix ops init critiques software logging
-
Great paper from Ben Maurer of Facebook in ACM Queue.
A "move-fast" mentality does not have to be at odds with reliability. To make these philosophies compatible, Facebook's infrastructure provides safety valves.
This is full of interesting techniques. * Rapidly deployed configuration changes: Make everybody use a common configuration system; Statically validate configuration changes; Run a canary; Hold on to good configurations; Make it easy to revert. * Hard dependencies on core services: Cache data from core services. Provide hardened APIs. Run fire drills. * Increased latency and resource exhaustion: Controlled Delay (based on the anti-bufferbloat CoDel algorithm -- this is really cool); Adaptive LIFO (last-in, first-out) for queue busting; Concurrency Control (essentially a form of circuit breaker). * Tools that Help Diagnose Failures: High-Density Dashboards with Cubism (horizon charts); What just changed? * Learning from Failure: the DERP (!) methodology,ben-maurer facebook reliability algorithms codel circuit-breakers derp failure ops cubism horizon-charts charts dependencies soa microservices uptime deployment configuration change-management
Tesla Autopilot mode is learning
This is really impressive, but also a little scary. Drivers driving the Tesla Model S are "phoning home" training data as they drive:
A Model S owner by the username Khatsalano kept a count of how many times he had to "rescue" (meaning taking control after an alert) his Model S while using the Autopilot on his daily commute. He counted 6 "rescues" on his first day, by the fourth day of using the system on his 23.5 miles commute, he only had to take control over once. Musk said that Model S owners could add ~1 million miles of new data every day, which is helping the company create "high precision maps".
Wonder if the data protection/privacy implications have been considered for EU use.autopilot tesla maps mapping training machine-learning eu privacy data-protection
-
For requesting a copy of an article that was legally obtained by a colleague from a paywalled source, Pazsowski found himself hit with around US$10,000-worth of damages. This completely disproportionate punishment for what is at most a minor case of copyright infringement is a perfect demonstration of where the anti-circumvention madness leads.
-
Add another one to the "yay for DST" pile. (also yay for AWS using PST/PDT as default internal timezone instead of UTC...)
utc timezones fail bugs aws aws-cli dst daylight-savings time
Google Cloud Platform HTTP/HTTPS Load Balancing
GCE's LB product is pretty nice -- HTTP/2 support, and a built-in URL mapping feature (presumably based on how Google approach that problem internally, I understand they take that approach). I'm hoping AWS are taking notes for the next generation of ELB, if that ever happens
elb gce google load-balancing http https spdy http2 urls request-routing ops architecture cloud
It's an Emulator, Not a Petting Zoo: Emu and Lambda
a Lambda emulator in Python, suitable for unit testing lambdas
Google tears Symantec a new one on its CA failure
Symantec are getting a crash course in how to conduct an incident post-mortem to boot:
More immediately, we are requesting of Symantec that they further update their public incident report with: A post-mortem analysis that details why they did not detect the additional certificates that we found. Details of each of the failures to uphold the relevant Baseline Requirements and EV Guidelines and what they believe the individual root cause was for each failure. We are also requesting that Symantec provide us with a detailed set of steps they will take to correct and prevent each of the identified failures, as well as a timeline for when they expect to complete such work. Symantec may consider this latter information to be confidential and so we are not requesting that this be made public.
google symantec ev ssl certificates ca security postmortems ops
Google is Maven Central's New Best Friend
google now mirroring Maven Central.
Apache Kafka, Purgatory, and Hierarchical Timing Wheels
In the new design, we use Hierarchical Timing Wheels for the timeout timer and DelayQueue of timer buckets to advance the clock on demand. Completed requests are removed from the timer queue immediately with O(1) cost. The buckets remain in the delay queue, however, the number of buckets is bounded. And, in a healthy system, most of the requests are satisfied before timeout, and many of the buckets become empty before pulled out of the delay queue. Thus, the timer should rarely have the buckets of the lower interval. The advantage of this design is that the number of requests in the timer queue is the number of pending requests exactly at any time. This allows us to estimate the number of requests need to be purged. We can avoid unnecessary purge operation of the watcher lists. As the result we achieve a higher scalability in terms of request rate with much better CPU usage.
algorithms timers kafka scheduling timing-wheels delayqueue queueing
Open-sourcing PalDB, a lightweight companion for storing side data
a new LinkedIn open source data store, for write-once/read-mainly side data, java, Apache licensed. RocksDB discussion: https://www.facebook.com/groups/rocksdb.dev/permalink/834956096602906/
linkedin open-source storage side-data data config paldb java apache databases
Twins denied driver's permit because DMV can't tell them apart
"The computer can recognize faces, a feature that comes in handy if somebody's is trying to get an illegal ID. It apparently is not programmed to detect twins." As Hilary Mason put it: "You do not want to be an edge case in this future we are building."
future grim bugs twins edge-cases coding fail dmv software via:hmason
The Okinawa missiles of October | Bulletin of the Atomic Scientists
'By Bordne's account, at the height of the Cuban Missile Crisis, Air Force crews on Okinawa were ordered to launch 32 missiles, each carrying a large nuclear warhead. Only caution and the common sense and decisive action of the line personnel receiving those orders prevented the launches—and averted the nuclear war that most likely would have ensued.'
okinawa nukes launch-codes pal cold-war cuban-missile-crisis history accidents ui security horror via:mattblaze
Amazon ECS CLI Tutorial - Amazon EC2 Container Service
super-basic ECS tutorial, using a docker-compose.yml to create a new ECS-managed service fleet
Net neutrality: EU votes in favour of Internet fast lanes and slow lanes | Ars Technica UK
:(
In the end, sheer political fatigue may have played a major part in undermining net neutrality in the EU. However, the battle is not quite over. As Anne Jellema, CEO of the Web Foundation, which was established by Berners-Lee in 2009, notes in her response to today's EU vote: "The European Parliament is essentially tossing a hot potato to the Body of European Regulators, national regulators and the courts, who will have to decide how these spectacularly unclear rules will be implemented. The onus is now on these groups to heed the call of hundreds of thousands of concerned citizens and prevent a two-speed Internet."
Analysing user behaviour - from histograms to random forests (PyData) at PyCon Ireland 2015 | Lanyrd
Swrve's own Dave Brodigan on game user-data analysis techniques:
The goal is to give the audience a roadmap for analysing user data using python friendly tools. I will touch on many aspects of the data science pipeline from data cleansing to building predictive data products at scale. I will start gently with pandas and dataframes and then discuss some machine learning techniques like kmeans and random forests in scikitlearn and then introduce Spark for doing it at scale. I will focus more on the use cases rather than detailed implementation. The talk will be informed by my experience and focus on user behaviour in games and mobile apps.
swrve talks user-data big-data spark hadoop machine-learning data-science
-
fast, modern, zero-conf load balancing HTTP(S) router managed by consul; serves 15k reqs/sec, in Go, from eBay
load-balancing consul http https routing ebay go open-source fabio
-
pretty conventional HTTP/1.1, WebSockets and HTTP/2 front-end services with modern Netty practices
RentTheRunway's Engineering Ladder
One of the best things about working at Amazon was having a clear, well-defined career progression, and it's something that's always been absent in startups. Career growth, levelling, and tech management is important, and also helps in hiring by providing clear levels. This is the RentTheRunway engineering ladder, Camille Fournier's team, which they open sourced back in March 2015
engineering hiring management career renttherunway camille-fournier amazon startups career-growth levelling ladder
How a criminal ring defeated the secure chip-and-PIN credit cards | Ars Technica
Ingenious --
The stolen cards were still considered evidence, so the researchers couldn't do a full tear-down or run any tests that would alter the data on the card, so they used X-ray scans to look at where the chip cards had been tampered with. They also analyzed the way the chips distributed electricity when in use and used read-only programs to see what information the cards sent to a Point of Sale (POS) terminal. According to the paper, the fraudsters were able to perform a man-in-the-middle attack by programming a second hobbyist chip called a FUN card to accept any PIN entry, and soldering that chip onto the card's original chip. This increased the thickness of the chip from 0.4mm to 0.7mm, "making insertion into a PoS somewhat uneasy but perfectly feasible," the researchers write. [....] The researchers explain that a typical EMV transaction involves three steps: card authentication, cardholder verification, and then transaction authorization. During a transaction using one of the altered cards, the original chip was allowed to respond with the card authentication as normal. Then, during card holder authentication, the POS system would ask for a user's PIN, the thief would respond with any PIN, and the FUN card would step in and send the POS the code indicating that it was ok to proceed with the transaction because the PIN checked out. During the final transaction authentication phase, the FUN card would relay the transaction data between the POS and the original chip, sending the issuing bank an authorization request cryptogram which the card issuer uses to tell the POS system whether to accept the transaction or not.
security chip-and-pin hacking pos emv transactions credit-cards debit-cards hardware chips pin fun-cards smartcards
How-to: Index Scanned PDFs at Scale Using Fewer Than 50 Lines of Code
using Spark, Tesseract, HBase, Solr and Leptonica. Actually pretty feasible
spark tesseract hbase solr leptonica pdfs scanning cloudera hadoop architecture
Existential Consistency: Measuring and Understanding Consistency at Facebook
The metric is termed ?(P)-consistency, and is actually very simple. A read for the same data is sent to all replicas in P, and ?(P)-consistency is defined as the frequency with which that read returns the same result from all replicas. ?(G)-consistency applies this metric globally, and ?(R)-consistency applies it within a region (cluster). Facebook have been tracking this metric in production since 2012.
facebook eventual-consistency consistency metrics papers cap distributed-computing
Holistic Configuration Management at Facebook
How FB push config changes from Git (where it is code reviewed, version controlled, and history tracked with strong auth) to Zeus (their Zookeeper fork) and from there to live production servers.
-
a high-performance multiple regex matching library. Hyperscan uses hybrid automata techniques to allow simultaneous matching of large numbers (up to tens of thousands) of regular expressions and for the matching of regular expressions across streams of data.
Via Tony Finchvia:fanf regexps regex dpi hyperscan dfa nfa hybrid-automata text-matching matching text strings streams
-
Hologram exposes an imitation of the EC2 instance metadata service on developer workstations that supports the [IAM Roles] temporary credentials workflow. It is accessible via the same HTTP endpoint to calling SDKs, so your code can use the same process in both development and production. The keys that Hologram provisions are temporary, so EC2 access can be centrally controlled without direct administrative access to developer workstations.
iam roles ec2 authorization aws adroll open-source cli osx coding dev
AWS re:Invent 2015 Video & Slide Presentation Links with Easy Index
Andrew Spyker's roundup:
my quick index of all re:Invent sessions. Please wait for a few days and I'll keep running the tool to fill in the index. It usually takes Amazon a few weeks to fully upload all the videos and slideshares.
Pretty definitive, full text descriptions of all sessions (and there are an awful lot of 'em).aws reinvent andrew-spyker scraping slides presentations ec2 video
(ARC308) The Serverless Company: Using AWS Lambda
Describing PlayOn! Sports' Lambda setup. Sounds pretty productionizable
Your Relative's DNA Could Turn You Into A Suspect
Familial DNA searching has massive false positives, but is being used to tag suspects:
The bewildered Usry soon learned that he was a suspect in the 1996 murder of an Idaho Falls teenager named Angie Dodge. Though a man had been convicted of that crime after giving an iffy confession, his DNA didn't match what was found at the crime scene. Detectives had focused on Usry after running a familial DNA search, a technique that allows investigators to identify suspects who don't have DNA in a law enforcement database but whose close relatives have had their genetic profiles cataloged. In Usry's case the crime scene DNA bore numerous similarities to that of Usry's father, who years earlier had donated a DNA sample to a genealogy project through his Mormon church in Mississippi. That project's database was later purchased by Ancestry, which made it publicly searchable—a decision that didn't take into account the possibility that cops might someday use it to hunt for genetic leads. Usry, whose story was first reported in The New Orleans Advocate, was finally cleared after a nerve-racking 33-day wait — the DNA extracted from his cheek cells didn't match that of Dodge's killer, whom detectives still seek. But the fact that he fell under suspicion in the first place is the latest sign that it's time to set ground rules for familial DNA searching, before misuse of the imperfect technology starts ruining lives.
dna familial-dna false-positives law crime idaho murder mormon genealogy ancestry.com databases biometrics privacy genes
Cluster benchmark: Scylla vs Cassandra
ScyllaDB (the C* clone in C++) is now actually looking promising -- still need more reassurance about its consistency/reliabilty side though
_What We Know About Spreadsheet Errors_ [paper]
As we will see below, there has long been ample evidence that errors in spreadsheets are pandemic. Spreadsheets, even after careful development, contain errors in one percent or more of all formula cells. In large spreadsheets with thousands of formulas, there will be dozens of undetected errors. Even significant errors may go undetected because formal testing in spreadsheet development is rare and because even serious errors may not be apparent.
business coding maths excel spreadsheets errors formulas error-rate
-
great post from Ross Duggan on avoiding developer burnout
How is NSA breaking so much crypto?
If a client and server are speaking Diffie-Hellman, they first need to agree on a large prime number with a particular form. There seemed to be no reason why everyone couldn't just use the same prime, and, in fact, many applications tend to use standardized or hard-coded primes. But there was a very important detail that got lost in translation between the mathematicians and the practitioners: an adversary can perform a single enormous computation to "crack" a particular prime, then easily break any individual connection that uses that prime. How enormous a computation, you ask? Possibly a technical feat on a scale (relative to the state of computing at the time) not seen since the Enigma cryptanalysis during World War II. Even estimating the difficulty is tricky, due to the complexity of the algorithm involved, but our paper gives some conservative estimates. For the most common strength of Diffie-Hellman (1024 bits), it would cost a few hundred million dollars to build a machine, based on special purpose hardware, that would be able to crack one Diffie-Hellman prime every year. Would this be worth it for an intelligence agency? Since a handful of primes are so widely reused, the payoff, in terms of connections they could decrypt, would be enormous. Breaking a single, common 1024-bit prime would allow NSA to passively decrypt connections to two-thirds of VPNs and a quarter of all SSH servers globally. Breaking a second 1024-bit prime would allow passive eavesdropping on connections to nearly 20% of the top million HTTPS websites. In other words, a one-time investment in massive computation would make it possible to eavesdrop on trillions of encrypted connections.
(via Eric)
AWS re:Invent 2015 | (CMP406) Amazon ECS at Coursera - YouTube
Coursera are running user-submitted code in ECS! interesting stuff about how they use Docker security/resource-limiting features, forking the ecs-agent code, to run user-submitted code. :O
coursera user-submitted-code sandboxing docker security ecs aws resource-limits ops
How both TCP and Ethernet checksums fail
At Twitter, a team had a unusual failure where corrupt data ended up in memcache. The root cause appears to have been a switch that was corrupting packets. Most packets were being dropped and the throughput was much lower than normal, but some were still making it through. The hypothesis is that occasionally the corrupt packets had valid TCP and Ethernet checksums. One "lucky" packet stored corrupt data in memcache. Even after the switch was replaced, the errors continued until the cache was cleared.
YA occurrence of this bug. When it happens, it tends to _really_ screw things up, because it's so rare -- we had monitoring for this in Amazon, and when it occurred, it overwhelmingly occurred due to host-level kernel/libc/RAM issues rather than stuff in the network. Amazon design principles were to add app-level checksumming throughout, which of course catches the lot.networking tcp ip twitter ethernet checksums packets memcached
Designing the Spotify perimeter
How Spotify use nginx as a frontline for their sites and services
scaling spotify nginx ops architecture ssl tls http frontline security
-
Supports Spotify -- totally getting one of these
Where do 'mama'/'papa' words come from?
The sounds came first — as experiments in vocalization — and parents adopted them as pet names for themselves. If you open your mouth and make a sound, it will probably be an open vowel like /a/ unless you move your tongue or lips. The easiest consonants are perhaps the bilabials /m/, /p/, and /b/, requiring no movement of the tongue, followed by consonants made by raising the front of the tongue: /d/, /t/, and /n/. Add a dash of reduplication, and you get mama, papa, baba, dada, tata, nana. That such words refer to people (typically parents or other guardians) is something we have imposed on the sounds and incorporated into our languages and cultures; the meanings don't inhere in the sounds as uttered by babies, which are more likely calls for food or attention.
-
'A fast build system for Docker images', open source, in Go, hooks into Github
England opens up 11TB of LiDAR data covering the entire country as open data
All 11 terabytes of our LIDAR data (that's roughly equivalent to 2,750,000 MP3 songs) will eventually be available through our new Open LIDAR portal under an Open Government Licence, allowing it to be used for any purpose. We hope that by giving free access to our data businesses and local communities will develop innovative solutions to benefit the environment, grow our thriving rural economy, and boost our world-leading food and farming industry. The possibilities are endless and we hope that making LIDAR data open will be a catalyst for new ideas and innovation.
Are you reading, Ordnance Survey Ireland?
SuperChief: From Apache Storm to In-House Distributed Stream Processing
Another sorry tale of Storm issues:
Storm has been successful at Librato, but we experienced many of the limitations cited in the Twitter Heron: Stream Processing at Scale paper and outlined here by Adrian Colyer, including: Inability to isolate, reason about, or debug performance issues due to the worker/executor/task paradigm. This led to building and configuring clusters specifically designed to attempt to mitigate these problems (i.e., separate clusters per topology, only running a worker per server.), which added additional complexity to development and operations and also led to over-provisioning. Ability of tasks to move around led to difficult to trace performance problems. Storm's work provisioning logic led to some tasks serving more Kafka partitions than others. This in turn created latency and performance issues that were difficult to reason about. The initial solution was to over-provision in an attempt to get a better hashing/balancing of work, but eventually we just replaced the work allocation logic. Due to Storm's architecture, it was very difficult to get a stack trace or heap dump because the processes that managed workers (Storm supervisor) would often forcefully kill a Java process while it was being investigated in this way. The propensity for unexpected and subsequently unhandled exceptions to take down an entire worker led to additional defensive verbose error handling everywhere. This nasty bug STORM-404 coupled with the aforementioned fact that a single exception can take down a worker led to several cascading failures in production, taking down entire topologies until we upgraded to 0.9.4. Additionally, we found the performance we were getting from Storm for the amount of money we were spending on infrastructure was not in line with our expectations. Much of this is due to the fact that, depending upon how your topology is designed, a single tuple may make multiple hops across JVMs, and this is very expensive. For example, in our time series aggregation topologies a single tuple may be serialized/deserialized and shipped across the wire 3-4 times as it progresses through the processing pipeline.
-
Librato's service discovery library using Zookeeper (so strongly consistent, but with the ZK downside that an AZ outage can stall service discovery updates region-wide)
zookeeper service-discovery librato java open-source load-balancing
Tech companies like Facebook not above the law, says Max Schrems
"Big companies didn't only rely on safe harbour: they also rely on binding corporate rules and standard contractual clauses. But it's interesting that the court decided the case on fundamental rights grounds: so it doesn't matter remotely what ground you transfer on, if that process is still illegal under 7 and 8 of charter, it can't be done."
Also:"Ireland has no interest in doing its job, and will continue not to, forever. Clearly it's an investment issue – but overall the policy is: we don't regulate companies here. The cost of challenging any of this in the courts is prohibitive. And the people don't seem to care."
:(ireland guardian max-schrems privacy surveillance safe-harbor eu us nsa dpc data-protection
After Bara: All your (Data)base are belong to us
Sounds like the CJEU's Bara decision may cause problems for the Irish government's wilful data-sharing:
Articles 10, 11 and 13 of Directive 95/46/EC of the European Parliament and of the Council of 24 October 1995, on the protection of individuals with regard to the processing of personal data and on the free movement of such data, must be interpreted as precluding national measures, such as those at issue in the main proceedings, which allow a public administrative body of a Member State to transfer personal data to another public administrative body and their subsequent processing, without the data subjects having been informed of that transfer or processing.
-
uses the techniques invented by the authors of Paris-traceroute to enumerate the paths of ECMP flow-based load balancing, but introduces a new technique for NAT detection.
handy. written by AWS SDE Andrea Barberio! -
'Seekable and Splittable Gzip', from eBay
Outage postmortem (2015-10-08 UTC) : Stripe: Help & Support
There was a breakdown in communication between the developer who requested the index migration and the database operator who deleted the old index. Instead of working on the migration together, they communicated in an implicit way through flawed tooling. The dashboard that surfaced the migration request was missing important context: the reason for the requested deletion, the dependency on another index's creation, and the criticality of the index for API traffic. Indeed, the database operator didn't have a way to check whether the index had recently been used for a query.
Good demo of how the Etsy-style chatops deployment approach would have helped avoid this risk.stripe postmortem outages databases indexes deployment chatops deploy ops
-
Wendy Grossman on where the Safe Harbor decision is leading.
One clause would require European companies to tell their relevant data protection authorities if they are being compelled to turn over data - even if they have been forbidden to disclose this under US law. Sounds nice, but doesn't mobilize the rock or soften the hard place, since companies will still have to pick a law to violate. I imagine the internal discussions there revolving around two questions: which violation is less likely to land the CEO in jail and which set of fines can we afford?
(via Simon McGarr)safe-harbor privacy law us eu surveillance wendy-grossman via:tupp_ed
-
bookmarking as a potential future addition to the back garden
Rebuilding Our Infrastructure with Docker, ECS, and Terraform
Good writeup of current best practices for a production AWS architecture
The Totally Managed Analytics Pipeline: Segment, Lambda, and Dynamo
notable mainly for the details of Terraform support for Lambda: that's a significant improvement to Lambda's production-readiness
aws pipelines data streaming lambda dynamodb analytics terraform ops
Gene patents probably dead worldwide following Australian court decision
The court based its reasoning on the fact that, although an isolated gene such as BRCA1 was "a product of human action, it was the existence of the information stored in the relevant sequences that was an essential element of the invention as claimed." Since the information stored in the DNA as a sequence of nucleotides was a product of nature, it did not require human action to bring it into existence, and therefore could not be patented.
Via Tony Finch.-
client-side 'service discovery and routing system for microservices' -- another Smartstack, then
python router smartstack baker-street microservices service-discovery routing load-balancing http
-
ugh, quite a bit of complexity here
Fuzzing Raft for Fun and Publication
Good intro to fuzz-testing a distributed system; I've had great results using similar approaches in unit tests
EC2 Spot Blocks for Defined-Duration Workloads
you can now launch Spot instances that will run continuously for a finite duration (1 to 6 hours). Pricing is based on the requested duration and the available capacity, and is typically 30% to 45% less than On-Demand.
The Surveillance Elephant in the Room…
Very perceptive post on the next steps for safe harbor, post-Schrems.
And behind that elephant there are other elephants: if US surveillance and surveillance law is a problem, then what about UK surveillance? Is GCHQ any less intrusive than the NSA? It does not seem so – and this puts even more pressure on the current reviews of UK surveillance law taking place. If, as many predict, the forthcoming Investigatory Powers Bill will be even more intrusive and extensive than current UK surveillance laws this will put the UK in a position that could rapidly become untenable. If the UK decides to leave the EU, will that mean that the UK is not considered a safe place for European data? Right now that seems the only logical conclusion – but the ramifications for UK businesses could be huge. [....] What happens next, therefore, is hard to foresee. What cannot be done, however, is to ignore the elephant in the room. The issue of surveillance has to be taken on. The conflict between that surveillance and fundamental human rights is not a merely semantic one, or one for lawyers and academics, it's a real one. In the words of historian and philosopher Quentin Skinner "the current situation seems to me untenable in a democratic society." The conflict over Safe Harbor is in many ways just a symptom of that far bigger problem. The biggest elephant of all.
ec cjeu surveillance safe-harbor schrems privacy europe us uk gchq nsa
ECJ ruling on Irish privacy case has huge significance
The only current way to comply with EU law, the judgment indicates, is to keep EU data within the EU. Whether those data can be safely managed within facilities run by US companies will not be determined until the US rules on an ongoing Microsoft case. Microsoft stands in contempt of court right now for refusing to hand over to US authorities, emails held in its Irish data centre. This case will surely go to the Supreme Court and will be an extremely important determination for the cloud business, and any company or individual using data centre storage. If Microsoft loses, US multinationals will be left scrambling to somehow, legally firewall off their EU-based data centres from US government reach.
(cough, Amazon)aws hosting eu privacy surveillance gchq nsa microsoft ireland
-
"@alexbfree @ThijsFeryn [ElasticSearch is] fine as long as data loss is acceptable. https://aphyr.com/posts/317-call-me-maybe-elasticsearch . We lose ~1% of all writes on average."
elasticsearch data-loss reliability data search aphyr jepsen testing distributed-systems ops
Daragh O'Brien on the CJEU judgement on Safe Harbor
Many organisations I've spoken to have had the cunning plan of adopting model contract clauses as their fall back position to replace their reliance on Safe Harbor. [....] The best that can be said for Model Clauses is that they haven't been struck down by the CJEU. Yet.
model-clauses cjeu eu europe safe-harbor us nsa surveillance privacy law
5 takeaways from the death of safe harbor – POLITICO
Reacting to the ruling, the [EC] stressed that data transfers between the U.S. and Europe can continue on the basis of other legal mechanisms. A lot rides on what steps the Commission and national data protection supervisors take in response. "It is crucial for legal certainty that the EC sends a clear signal," said Nauwelaerts. That could involve providing a timeline for concluding an agreement with U.S. authorities, together with a commitment from national data protection authorities not to block data transfers while negotiations are on-going, he explained.
The New InfluxDB Storage Engine: A Time Structured Merge Tree
The new engine has similarities with LSM Trees (like LevelDB and Cassandra's underlying storage). It has a write ahead log, index files that are read only, and it occasionally performs compactions to combine index files. We're calling it a Time Structured Merge Tree because the index files keep contiguous blocks of time and the compactions merge those blocks into larger blocks of time. Compression of the data improves as the index files are compacted. Once a shard becomes cold for writes it will be compacted into as few files as possible, which yield the best compression.
influxdb storage lsm-trees leveldb tsm-trees data-structures algorithms time-series tsd compression
Marvin.ie: Order Takeaway Food Online
new Dublin delivery service takes Bitcoin?!
qp tries: smaller and faster than crit-bit tries
interesting new data structure from Tony Finch. "Some simple benchmarks say qp tries have about 1/3 less memory overhead and are about 10% faster than crit-bit tries."
crit-bit popcount bits bitmaps tries data-structures via:fanf qp-tries crit-bit-tries hacks memory
Schneier on Automatic Face Recognition and Surveillance
When we talk about surveillance, we tend to concentrate on the problems of data collection: CCTV cameras, tagged photos, purchasing habits, our writings on sites like Facebook and Twitter. We think much less about data analysis. But effective and pervasive surveillance is just as much about analysis. It's sustained by a combination of cheap and ubiquitous cameras, tagged photo databases, commercial databases of our actions that reveal our habits and personalities, and -- most of all -- fast and accurate face recognition software. Don't expect to have access to this technology for yourself anytime soon. This is not facial recognition for all. It's just for those who can either demand or pay for access to the required technologies -- most importantly, the tagged photo databases. And while we can easily imagine how this might be misused in a totalitarian country, there are dangers in free societies as well. Without meaningful regulation, we're moving into a world where governments and corporations will be able to identify people both in real time and backwards in time, remotely and in secret, without consent or recourse. Despite protests from industry, we need to regulate this budding industry. We need limitations on how our images can be collected without our knowledge or consent, and on how they can be used. The technologies aren't going away, and we can't uninvent these capabilities. But we can ensure that they're used ethically and responsibly, and not just as a mechanism to increase police and corporate power over us.
privacy regulation surveillance bruce-schneier faces face-recognition machine-learning ai cctv photos
-
China just introduced a universal credit score, where everybody is measured as a number between 350 and 950. But this credit score isn't just affected by how well you manage credit – it also reflects how well your political opinions are in line with Chinese official opinions, and whether your friends' are, too.
Measuring using online mass surveillance, naturally. This may be the most dystopian thing I've heard in a while....via:raycorrigan dystopia china privacy mass-surveillance politics credit credit-score loans opinions
Brand New Retro – The Book, November 2015
YESSSS. Joe and Brian have delivered -- going to be giving a lot of copies of this for xmas ;)
brand-new-retro blogs friends retro history dublin ireland books toget
-
your command line environment in the [Google] Cloud. This feature enables you to connect to a shell environment on a virtual machine, pre-loaded with the tools you need to easily run commands to develop, deploy and manage your projects. Currently, Cloud Shell is an f1-micro Google Compute Engine machine that exposes a Debian-based development environment. You are also assigned 5 GB of standard persistent disk space as the home disk so you can store files between sessions.
It's also free. This is a great idea -- handy both for beginners getting to grips with GoogCloud and for experts looking for a quite dev env to hack with. I wish AWS had something similar. Amaro: A Bittersweet Obsession - Food & Wine
"A Neapolitan-American friend of mine, who's in his mid-fifties, fondly remembers how his mother used to serve him an espresso with Fernet Branca and an egg yolk every morning before he went off to elementary school."
-
come recommended by http://gearmoose.com/the-ten-best-minimalist-wallets-a-recap/ , looks pretty nice
Notes on Startup Engineering Management for Young Bloods
Below is a list of some lessons I've learned as an startup engineering manager that are worth being told to a new manager. Some are subtle, and some are surprising, and this being human beings, some are inevitably controversial. This list is for the new head of engineering to guide their thinking about the job they are taking on. It's not comprehensive, but it's a good beginning. The best characteristic of this list is that it focuses on social problems with little discussion of technical problems a manager may run into. The social stuff is usually the hardest part of any software developer's job, and of course this goes triply for engineering managers.
Further reading on just culture and blameless post mortems
Some bookmarks around post-mortem activity
post-mortems culture etsy rafe-colburn rc3 john-allspaw ops coes
Han Sung: Probably the Best Korean Food in Dublin
Han Sung is bizarrely located in the back of an Asian supermarket just off the Millennium Walk on Great Strand Street. [...] You'd see this a lot in Korea, I ask, a restaurant in the back of a supermarket? Not really, no, he says.
Behold: The Ultimate Crowdsourced Map of Punny Businesses in America | Atlas Obscura
"Spex in the City", "Fidler on the Tooth", "Sight For Four Eyes", "Fried Egg I'm In Love", "Lice Knowing You" and many more
-
this is quite nice. PipelineDB allows direct hookup of a Kafka stream, and will ingest durably and reliably, and provide SQL views computed over a sliding window of the stream.
logging sql kafka pipelinedb streaming sliding-window databases search querying
the impact of the economic crisis on public funding for universities in Europe
Ireland leading the pack with a drop of funding by 20% :(
universities ireland ucd tcd dcu funding public-funding europe history downturn
CurrencyFair P2P International Money Transfers
recommended by Paul Hickey
via:phickey money money-transfer currency currency-conversion tools recommendations
How the banks ignored the lessons of the crash
First of all, banks could be chopped up into units that can safely go bust – meaning they could never blackmail us again. Banks should not have multiple activities going on under one roof with inherent conflicts of interest. Banks should not be allowed to build, sell or own overly complex financial products – clients should be able to comprehend what they buy and investors understand the balance sheet. Finally, the penalty should land on the same head as the bonus, meaning nobody should have more reason to lie awake at night worrying over the risks to the bank's capital or reputation than the bankers themselves. You might expect all major political parties to have come out by now with their vision of a stable and productive financial sector. But this is not what has happened.
The price of the Internet of Things will be a vague dread of a malicious world
So the fact is that our experience of the world will increasingly come to reflect our experience of our computers and of the internet itself (not surprisingly, as it'll be infused with both). Just as any user feels their computer to be a fairly unpredictable device full of programs they've never installed doing unknown things to which they've never agreed to benefit companies they've never heard of, inefficiently at best and actively malignant at worst (but how would you now?), cars, street lights, and even buildings will behave in the same vaguely suspicious way. Is your self-driving car deliberately slowing down to give priority to the higher-priced models? Is your green A/C really less efficient with a thermostat from a different company, or it's just not trying as hard? And your tv is supposed to only use its camera to follow your gestural commands, but it's a bit suspicious how it always offers Disney downloads when your children are sitting in front of it. None of those things are likely to be legal, but they are going to be profitable, and, with objects working actively to hide them from the government, not to mention from you, they'll be hard to catch.
culture bots criticism ieet iot internet-of-things law regulation open-source appliances
excellent offline mapping app MAPS.ME goes open source
"MAPS.ME is an open source cross-platform offline maps application, built on top of crowd-sourced OpenStreetMap data. It was publicly released for iOS and Android."
Eircode cost the Irish government EUR38m
The C&AG has said it is not clear that the €38m scheme will achieve the data-matching benefits the Government had hoped.
Well, that's putting it mildly.Let a 1,000 flowers bloom. Then rip 999 of them out by the roots
The Twitter tech-debt story.
Somewhere along the way someone decided that it would be easier to convert the Birdcage to use Pants which had since learned how to build Scala and to deal with a maven-style layout. However at some point prior Pants been open sourced in throw it over the wall fashion and picked up by a few engineers at other companies, such as Square and Foursquare and moved forward. In the meantime, again because there weren't enough people who's job it was to take care of these things, Science was still on the original internally developed version and had in fact evolved independently of the open source version. However by the time we wanted to move Birdcage onto Pants, the open source version had moved ahead so that's the one the Birdcage folks chose.
(cries)tech-debt management twitter productivity engineering monorepo build-systems war-stories dev
-
Amazing. This is what happens when embedded software engineers make a UI, in my experience
embedded-software ui ux design graphics windows the-horror omgwtf atms
EPA opposed rules that would have exposed VW's cheating
[...] Two months ago, the EPA opposed some proposed measures that would help potentially expose subversive code like the so-called "defeat device" software VW allegedly used by allowing consumers and researchers to legally reverse-engineer the code used in vehicles. EPA opposed this, ironically, because the agency felt that allowing people to examine the software code in vehicles would potentially allow car owners to alter the software in ways that would produce more emissions in violation of the Clean Air Act. The issue involves the 1998 Digital Millennium Copyright Act (DCMA), which prohibits anyone from working around "technological protection measures" that limit access to copyrighted works. The Library of Congress, which oversees copyrights, can issue exemptions to those prohibitions that would make it legal, for example, for researchers to examine the code to uncover security vulnerabilities.
dmca volkswagen vw law code open-source air-quality diesel cheating regulation us-politics
From Radio to Porn, British Spies Track Web Users' Online Identities
Inside KARMA POLICE, GCHQ's mass-surveillance operation aimed to record the browsing habits of "every visible user on the internet", including UK-to-UK internal traffic. more details on the other GCHQ mass surveillance projects at https://theintercept.com/gchq-appendix/
surveillance gchq security privacy law uk ireland karma-police snooping
Streaming will soon pass traditional TV - Tech Insider
the percentage of people who say they stream video from services like Netflix, YouTube, and Hulu each day has increased dramatically over the last five years, from about 30% in 2010 to more than 50% this year. During the same period, the percentage of people who say they watch traditional TV [...] has dropped by about 10%. When the beige line surpasses the purple line [looks like 2016], it will mean that more people are streaming each day than are watching traditional TV.
Is there a CAP theorem for Durability?
Marc Brooker with another thought-provoking blogpost
databases storage marc-brooker cap-theorem cap durability pacelc nosql
-
(via Aman)
via:akohli graphics ascii-art ascii visualization text boxes diagrams
Scale it to Billions — What They Don't Tell you in the Cassandra README
large-scale C* tips
Introduction to HDFS Erasure Coding in Apache Hadoop
How Hadoop did EC. Erasure Coding support ("HDFS-EC") is set to be released in Hadoop 3.0 apparently
erasure-coding reed-solomon algorithms hadoop hdfs cloudera raid storage
-
some details on Netflix's Chaos Monkey, Chaos Kong and other aspects of their availability/failover testing
architecture aws netflix ops chaos-monkey chaos-kong testing availability failover ha
-
Træf?k is a modern HTTP reverse proxy and load balancer made to deploy microservices with ease. It supports several backends (Docker , Mesos/Marathon, Consul, Etcd, Rest API, file...) to manage its configuration automatically and dynamically.
Hot-reloading is notably much easier than with nginx/haproxy. -
a proxy that mucks with your system and application context, operating at Layers 4 and 7, allowing you to simulate common failure scenarios from the perspective of an application under test; such as an API or a web application. If you are building a distributed system, Muxy can help you test your resilience and fault tolerance patterns.
proxy distributed testing web http fault-tolerance failure injection tcp delay resilience error-handling
Petabyte-Scale Data Pipelines with Docker, Luigi and Elastic Spot Instances — AdRoll
nice approach
-
a tool which simplifies tracing and testing of Java programs. Byteman allows you to insert extra Java code into your application, either as it is loaded during JVM startup or even after it has already started running. The injected code is allowed to access any of your data and call any application methods, including where they are private. You can inject code almost anywhere you want and there is no need to prepare the original source code in advance nor do you have to recompile, repackage or redeploy your application. In fact you can remove injected code and reinstall different code while the application continues to execute. The simplest use of Byteman is to install code which traces what your application is doing. This can be used for monitoring or debugging live deployments as well as for instrumenting code under test so that you can be sure it has operated correctly. By injecting code at very specific locations you can avoid the overheads which often arise when you switch on debug or product trace. Also, you decide what to trace when you run your application rather than when you write it so you don't need 100% hindsight to be able to obtain the information you need.
Henry Robinson on testing and fault discovery in distributed systems
'Let's talk about finding bugs in distributed systems for a bit. These chaos monkey-style fault testing systems are all well and good, but by being application independent they're a very blunt instrument. Particularly they make it hard to search the fault space for bugs in a directed manner, because they don't 'know' what the system is doing. Application-aware scripting of faults in a dist. systems seems to be rarely used, but allows you to directly stress problem areas. For example, if a bug manifests itself only when one RPC returns after some timeout, hard to narrow that down with iptables manipulation. But allow a script to hook into RPC invocations (and other trace points, like DTrace's probes), and you can script very specific faults. That way you can simulate cross-system integration failures, *and* write reproducible tests for the bugs they expose! Anyhow, I've been doing this in Impala, and it's been very helpful. Haven't seen much evidence elsewhere.'
henry-robinson testing fault-discovery rpc dtrace tracing distributed-systems timeouts chaos-monkey impala
The Best Bourbon Cocktail You've Never Heard Of
The "Paper Plane", by Sam Ross of Chicago's "Violet Hour": .75 oz Bourbon .75 oz Aperol .75 oz Amaro Nonino .75 oz Fresh lemon juice ice-filled shaker, shake, strain.
-
C++ high-performance app framework; 'currently focused on high-throughput, low-latency I/O intensive applications.' Scylla (Cassandra-compatible NoSQL store) is written in this.
c++ opensource performance framework scylla seastar latency linux shared-nothing multicore
How VW tricked the EPA's emissions testing system
In July 2015, CARB did some follow up testing and again the cars failed—the scrubber technology was present, but off most of the time. How this happened is pretty neat. Michigan's Stefanopolou says computer sensors monitored the steering column. Under normal driving conditions, the column oscillates as the driver negotiates turns. But during emissions testing, the wheels of the car move, but the steering wheel doesn't. That seems to have have been the signal for the "defeat device" to turn the catalytic scrubber up to full power, allowing the car to pass the test. Stefanopolou believes the emissions testing trick that VW used probably isn't widespread in the automotive industry. Carmakers just don't have many diesels on the road. And now that number may go down even more.
Depressing stuff -- but at least they think VW's fraud wasn't widespread.fraud volkswagen vw diesel emissions air-quality epa carb catalytic-converters testing
EU court adviser: data-share deal with U.S. is invalid | Reuters
The Safe Harbor agreement does not do enough to protect EU citizen's private information when it reached the United States, Yves Bot, Advocate General at the European Court of Justice (ECJ), said. While his opinions are not binding, they tend to be followed by the court's judges, who are currently considering a complaint about the system in the wake of revelations from ex-National Security Agency contractor Edward Snowden of mass U.S. government surveillance.
safe-harbor law eu ec ecj snowden surveillance privacy us data max-schrems
Summary of the Amazon DynamoDB Service Disruption and Related Impacts in the US-East Region
Painful to read, but: tl;dr: monitoring oversight, followed by a transient network glitch triggering IPC timeouts, which increased load due to lack of circuit breakers, creating a cascading failure
aws postmortem outages dynamodb ec2 post-mortems circuit-breakers monitoring
What Happens Next Will Amaze You
Maciej Ceglowski's latest talk, on ads, the web, Silicon Valley and government:
'I went to school with Bill. He's a nice guy. But making him immortal is not going to make life better for anyone in my city. It will just exacerbate the rent crisis.'
talks slides funny ads advertising internet web privacy surveillance maciej silicon-valley
Frame of Reference and Roaring Bitmaps
interesting performance-oriented algorithm tweak from Elastic/Lucene
lucene elasticsearch performance optimization roaring-bitmaps bitmaps frame-of-reference integers algorithms
Uber Goes Unconventional: Using Driver Phones as a Backup Datacenter - High Scalability
Initially I thought they were just tracking client state on the phone, but it actually sounds like they're replicating other users' state, too. Mad stuff! Must cost a fortune in additional data transfer costs...
Brotli: a new compression algorithm for the internet from Google
While Zopfli is Deflate-compatible, Brotli is a whole new data format. This new format allows us to get 20–26% higher compression ratios over Zopfli. In our study 'Comparison of Brotli, Deflate, Zopfli, LZMA, LZHAM and Bzip2 Compression Algorithms' we show that Brotli is roughly as fast as zlib's Deflate implementation. At the same time, it compresses slightly more densely than LZMA and bzip2 on the Canterbury corpus. The higher data density is achieved by a 2nd order context modeling, re-use of entropy codes, larger memory window of past data and joint distribution codes. Just like Zopfli, the new algorithm is named after Swiss bakery products. Brötli means 'small bread' in Swiss German.
brotli zopfli deflate gzip compression algorithms swiss google
-
'The key thing about Ubiquiti gear is the high quality radios and antennas. It just seems much more reliable than most consumer WiFi gear. Their airOS firmware is good too, it's a bit complicated to set up but very capable and flexible. And in addition to normal 802.11n or 802.11ac they also have an optional proprietary TDMA protocol called airMax that's designed for serving several long haul links from a single basestation. They're mostly marketing to business customers but the equipment is sold retail and well documented for ordinary nerds to figure out.'
ubiquiti wifi wireless 802.11 via:nelson ethernet networking prosumer hardware wan
-
a specialized packet sniffer designed for displaying and logging HTTP traffic. It is not intended to perform analysis itself, but to capture, parse, and log the traffic for later analysis. It can be run in real-time displaying the traffic as it is parsed, or as a daemon process that logs to an output file. It is written to be as lightweight and flexible as possible, so that it can be easily adaptable to different applications.
via Eoin Brazilvia:eoinbrazil httpry http networking tools ops testing tcpdump tracing
ustwo Reimagines the In-Car Cluster
Designers behind the cult mobile game, Monument Valley, take on the legacy-bound in-car UI
-
'It's very easy: So long as you don't hear "The Little Drummer Boy," you're a contender. As soon as you hear it on the radio, on TV, in a store, wherever, you're out.'
Geographically-accurate version of the London underground map
as Boing Boing says: 'London's subway system switched early to an abstract map (PDF), and it became a legendary work of design. It just published an internally-used geographic version of map (PDF), however, for the first time in a century—and it's awesome.'
london maps mapping geography accuracy pdf subway underground
Critiki's top 10 tiki bars in the world
not a one in Europe, of course! I need to hit up one of these sometime
What is the fastest way to clone a git repository over a fast network connection? - Stack Overflow
"git bundle create" -- neat trick
-
a regex-based, Turing-complete programming language. It's main feature is taking some text via standard input and repeatedly applying regex operations to it (e.g. matching, splitting, and most of all replacing). Under the hood, it uses .NET's regex engine, which means that both the .NET flavour and the ECMAScript flavour are available.
Reminscent of sed(1); see http://codegolf.stackexchange.com/a/58166 for an example Retina programretina regexps regexes regular-expressions coding hacks dot-net languages
Time on multi-core, multi-socket servers
Nice update on the state of System.currentTimeMillis() and System.nanoTime() in javaland. Bottom line: both are non-monotonic nowadays:
The conclusion I've reached is that except for the special case of using nanoTime() in micro benchmarks, you may as well stick to currentTimeMillis() —knowing that it may sporadically jump forwards or backwards. Because if you switched to nanoTime(), you don't get any monotonicity guarantees, it doesn't relate to human time any more —and may be more likely to lead you into writing code which assumes a fast call with consistent, monotonic results.
java time monotonic sequencing nanotime timers jvm multicore distributed-computing
Anatomy of a Modern Production Stack
Interesting post, but I think it falls into a common trap for the xoogler or ex-Amazonian -- assuming that all the BigCo mod cons are required to operate, when some are luxuries than can be skipped for a few years to get some real products built
architecture ops stack docker containerization deployment containers rkt coreos prod monitoring xooglers
How We Use AWS Lambda for Rapidly Intensifying Workloads · CloudSploit
impressive -- pretty much the entire workload is run from Lambda here
Introducing the Software Testing Cupcake (Anti-Pattern)
good post on the risks of overweighting towards manual testing rather than low-level automated tests (via Tony Byrne)
Kate Heddleston: How Our Engineering Environments Are Killing Diversity
'[There are] several problem areas for [diversity in] engineering environments and ways to start fixing them. The problems we face aren't devoid of solutions; there are a lot of things that companies, teams, and individuals can do to fix problems in their work environment. For the month of March, I will be posting detailed articles about the problem areas I will cover in my talk: argument cultures, feedback, promotions, employee on-boarding, benefits, safety, engineering process, and environment adaptation.' via Baron Schwartz.
via:xaprb culture tech diversity sexism feminism engineering work workplaces feedback
-
'Heavily tinted blue paintings form space stations, spacesuits, and rockets just after blast. Michael Kagan paints these large-scale works to celebrate the man-made object—machinery that both protects and holds the possibility of instantly killing those that operate the equipment from the inside. To paint the large works, Kagan utilizes an impasto technique with thick strokes that are deliberate and unique, showing an aggression in his application of oil paint on linen. The New York-based artist focuses on iconic images in his practice, switching back and forth between abstract and representational styles. "The painting is finished when it can fall apart and come back together depending on how it is read and the closeness to the work," said Kagan about his work. "Each painting is an image, a snapshot, a flash moment, a quick read that is locked into memory by the iconic silhouettes."' Via http://www.thisiscolossal.com/2015/08/michael-kagens-space-paintings/
-
I'm assuming, if you are on the Internet and reading kind of a nerdy blog, that you know what Unicode is. At the very least, you have a very general understanding of it — maybe "it's what gives us emoji". That's about as far as most people's understanding extends, in my experience, even among programmers. And that's a tragedy, because Unicode has a lot of… ah, depth to it. Not to say that Unicode is a terrible disaster — more that human language is a terrible disaster, and anything with the lofty goals of representing all of it is going to have some wrinkles. So here is a collection of curiosities I've encountered in dealing with Unicode that you generally only find out about through experience. Enjoy.
unicode characters encoding emoji utf-8 utf-16 utf mysql text
httpbin(1): HTTP Client Testing Service
Testing an HTTP Library can become difficult sometimes. RequestBin is fantastic for testing POST requests, but doesn't let you control the response. This exists to cover all kinds of HTTP scenarios. Additional endpoints are being considered.
-
amazing slideshow/WebGL demo talking about graphics programming, its maths, and GPUs
maths graphics webgl demos coding algorithms slides tflops gpus
'I wish to register a complaint': know your consumer rights before the fight
Conor Pope on the basics of consumer law -- and how to complain -- in Ireland
-
an object pooling library for Java. Use it to recycle objects that are expensive to create. The library will take care of creating and destroying your objects in the background. Stormpot is very mature, is used in production, and has done over a trillion claim-release cycles in testing. It is faster and scales better than any competing pool.
Apache-licensed, and extremely fast: https://medium.com/@chrisvest/released-stormpot-2-4-eeab4aec86d0java stormpot object-pooling object-pools pools allocation gc open-source apache performance
Evolution of Babbel's data pipeline on AWS: from SQS to Kinesis
Good "here's how we found it" blog post:
Our new data pipeline with Kinesis in place allows us to plug new consumers without causing any damage to the current system, so it's possible to rewrite all Queue Workers one by one and replace them with Kinesis Workers. In general, the transition to Kinesis was smooth and there were not so tricky parts. Another outcome was significantly reduced costs – handling almost the same amount of data as SQS, Kinesis appeared to be many times cheaper than SQS.
aws kinesis kafka streaming data-pipelines streams sqs queues architecture kcl
You're probably wrong about caching
Excellent cut-out-and-keep guide to why you should add a caching layer. I've been following this practice for the past few years, after I realised that #6 (recovering from a failed cache is hard) is a killer -- I've seen a few large-scale outages where a production system had gained enough scale that it required a cache to operate, and once that cache was damaged, bringing the system back online required a painful rewarming protocol. Better to design for the non-cached case if possible.
architecture caching coding design caches ops production scalability
The Alternative Universe Of Soviet Arcade Games
Unlike machines in the West, every single machine that was produced during Soviet-era Russia had to align with Marxist ideology. [...] The most popular games were created to teach hand-eye coordination, reaction speed, and logical, focused thinking. Not unlike many American games, these games were influenced by military training, crafted to teach and instill patriotism for the state by making the human body better, stronger, and more willful. It also means no high scores, no adrenaline rushes, or self-serving feather-fluffing as you add your hard-earned initials to the list of the best. In Communist Russia, there was no overt competition.
high-scores communism russia cccp ussr arcade-games games history
Large Java HashMap performance overview
Large HashMap overview: JDK, FastUtil, Goldman Sachs, HPPC, Koloboke, Trove – January 2015 version
java performance hashmap hashmaps optimization fastutil hppc jdk koloboke trove data-structures
-
Is it too late to replace Eircode?
Addresses are hard. Who can remember street addresses or latitude/longitude pairs? You could do much better with three totally random English words, but then there's that pesky language barrier. No system is perfect, except for emoji.
eircode maps parody via:nelson location geocoding mapping pile-of-poo
Real Time Analytics With Spark Streaming and Cassandra
...and Kafka
spark-streaming kafka analytics cassandra architecture data batch
Improvements to Kafka integration of Spark Streaming
looks decent as an approach
Diffy: Testing services without writing tests
Play requests against 2 versions of a service. A fair bit more complex than simply replaying logged requests, which took 10 lines of a shell script last time I did it
Gmail supports animated emoji in e-mail subjects
Currently only used in spam, naturally. (via Hilary Mason)
-
The Algorithmist is a resource dedicated to anything algorithms - from the practical realm, to the theoretical realm. There are also links and explanation to problemsets.
A wiki for algorithms. Not sure if this is likely to improve on Wikipedia, which of course covers the same subject matter quite well, though -
analyzes Spot price history to help you determine a bid price that suits your needs.
What Are the Worst Airports in the World?
this is a great resource when picking a stopover for a 2-stop flight. Pity "best kids play area" isn't a criterion
airports comparison via:boingboing flying travel ranking world skytrax
Using Samsung's Internet-Enabled Refrigerator for Man-in-the-Middle Attacks
Whilst the fridge implements SSL, it FAILS to validate SSL certificates, thereby enabling man-in-the-middle attacks against most connections. This includes those made to Google's servers to download Gmail calendar information for the on-screen display. So, MITM the victim's fridge from next door, or on the road outside and you can potentially steal their Google credentials.
The Internet of Insecure Things strikes again.iot security fridges samsung fail mitm ssl tls google papers defcon
Malware infecting jailbroken iPhones stole 225,000 Apple account logins | Ars Technica
KeyRaider, as the malware family has been dubbed, is distributed through a third-party repository of Cydia, which markets itself as an alternative to Apple's official App Store. Malicious code surreptitiously included with Cydia apps is creating problems for people in China and at least 17 other countries, including France, Russia, Japan, and the UK. Not only has it pilfered account data for 225,941 Apple accounts, it has also disabled some infected phones until users pay a ransom, and it has made unauthorized charges against some victims' accounts.
Ouch. Not a good sign for Cydiacydia apple security exploits jailbreaking ios iphone malware keyraider china
-
'a simple command line tool that turns your CLI tools into web applications'
-
a file system that stores all its data online using storage services like Google Storage, Amazon S3, or OpenStack. S3QL effectively provides a hard disk of dynamic, infinite capacity that can be accessed from any computer with internet access running Linux, FreeBSD or OS-X. S3QL is a standard conforming, full featured UNIX file system that is conceptually indistinguishable from any local file system. Furthermore, S3QL has additional features like compression, encryption, data de-duplication, immutable trees and snapshotting which make it especially suitable for online backup and archival. S3QL is designed to favor simplicity and elegance over performance and feature-creep. Care has been taken to make the source code as readable and serviceable as possible. Solid error detection and error handling have been included from the very first line, and S3QL comes with extensive automated test cases for all its components.
3 Lessons From The Amazon Takedown - Fortune
They are: The leaders we admire aren't always that admirable; Economic performance and costs trump employee well-being; and people participate in and rationalize their own subjugation. 'In the end, "Amazonians" are not that different from other people in their psychological dynamics. Their company is just a more extreme case of what many other organizations regularly do. And most importantly, let's locate the problem, if there is one, and its solution where it most appropriately belongs—not with a CEO who is greatly admired (and wealthy beyond measure) running a highly admired company, but with a society where money trumps human well-being and where any price, maybe even lives, is paid for status and success.' (via Lean)
amazon work work-life-balance life us fortune via:ldoody ceos employment happiness
What does it take to make Google work at scale? [slides]
50-slide summary of Google's stack, compared vs Facebook, Yahoo!, and open-source-land, with the odd interesting architectural insight
google architecture slides scalability bigtable spanner facebook gfs storage
Scaling Analytics at Amplitude
Good blog post on Amplitude's lambda architecture setup, based on S3 and a custom "real-time set database" they wrote themselves. antirez' comment from a Redis angle on the set database: http://antirez.com/news/92 HN thread: https://news.ycombinator.com/item?id=10118413
lambda-architecture analytics via:hn redis set-storage storage databases architecture s3 realtime
-
toxy is a fully programmatic and hackable HTTP proxy to simulate server failure scenarios and unexpected network conditions. It was mainly designed for fuzzing/evil testing purposes, when toxy becomes particularly useful to cover fault tolerance and resiliency capabilities of a system, especially in service-oriented architectures, where toxy may act as intermediate proxy among services. toxy allows you to plug in poisons, optionally filtered by rules, which essentially can intercept and alter the HTTP flow as you need, performing multiple evil actions in the middle of that process, such as limiting the bandwidth, delaying TCP packets, injecting network jitter latency or replying with a custom error or status code.
toxy proxies proxy http mitm node.js soa network failures latency slowdown jitter bandwidth tcp
Drone Oversight Is Coming to Construction Sites
Grim Meathook Future
grim-meathook-future drones work panopticon future sacramento building-sites
-
Open source security team has had enough of embedded-systems vendors taking the piss with licensing:
This announcement is our public statement that we've had enough. Companies in the embedded industry not playing by the same rules as every other company using our software violates users' rights, misleads users and developers, and harms our ability to continue our work. Though I've only gone into depth in this announcement on the latest trademark violation against us, our experience with two GPL violations over the previous year have caused an incredible amount of frustration. These concerns are echoed by the complaints of many others about the treatment of the GPL by the embedded Linux industry in particular over many years. With that in mind, today's announcement is concerned with the future availability of our stable series of patches. We decided that it is unfair to our sponsors that the above mentioned unlawful players can get away with their activity. Therefore, two weeks from now, we will cease the public dissemination of the stable series and will make it available to sponsors only. The test series, unfit in our view for production use, will however continue to be available to the public to avoid impact to the Gentoo Hardened and Arch Linux communities. If this does not resolve the issue, despite strong indications that it will have a large impact, we may need to resort to a policy similar to Red Hat's, described here or eventually stop the stable series entirely as it will be an unsustainable development model.
culture gpl linux opensource security grsecurity via:nelson gentoo arch-linux gnu
London Calling: Two-Factor Authentication Phishing From Iran
some rather rudimentary anti-2FA attempts, presumably from Iranian security services
Vegemite May Power The Electronics Of The Future
Professor Marc in het Panhuis at the ARC Centre of Excellence for Electromaterials Science figured out that you can 3D print the paste and use it to carry current, effectively creating Vegemite bio-wires. What does this mean? Soon you can run electricity through your food. "The iconic Australian Vegemite is ideal for 3D printing edible electronics," said the professor. "It contains water so it's not a solid and can easily be extruded using a 3D printer. Also, it's salty, so it conducts electricity."
I'm sure the same applies for Marmite...Beoir.org Community - Recent Attack on McGargles
bizarre conspiracy theory going around about McGargles microbrewery being owned by Molson in an "astroturf craft beer" operation -- they apparently were set up by a bunch of ex-Molson employees. Their beer is getting stickered in off-licenses. Mental!
beer craft-beer ireland mcgargles conspiracy-theories bizarre beoir
Mining High-Speed Data Streams: The Hoeffding Tree Algorithm
This paper proposes a decision tree learner for data streams, the Hoeffding Tree algorithm, which comes with the guarantee that the learned decision tree is asymptotically nearly identical to that of a non-incremental learner using infinitely many examples. This work constitutes a significant step in developing methodology suitable for modern 'big data' challenges and has initiated a lot of follow-up research. The Hoeffding Tree algorithm has been covered in various textbooks and is available in several public domain tools, including the WEKA Data Mining platform.
hoeffding-tree algorithms data-structures streaming streams cep decision-trees ml learning papers
Chinese scammers are now using Stingray tech to SMS-phish
A Stingray-style false GSM base station, hidden in a backpack; presumably they detect numbers in the vicinity, and SMS-spam those numbers with phishing messages. Reportedly the scammers used this trick in "Guangzhou, Zhuhai, Shenzhen, Changsha, Wuhan, Zhengzhou and other densely populated cities". Dodgy machine translation:
March 26, Zhengzhou police telecommunications fraud cases together, for the first time seized a small backpack can hide pseudo station equipment, and arrested two suspects. Yesterday, the police informed of this case, to remind the general public to pay attention to prevention. "I am the landlord, I changed number, please rent my wife hit the bank card, card number ×××, username ××." Recently, Jiefang Road, Zhengzhou City Public Security Bureau police station received a number of cases for investigation brigade area of ??the masses police said, frequently received similar phone scam messages. Alarm, the police investigators to determine: the suspect may be in the vicinity of twenty-seven square, large-scale use of mobile pseudo-base release fraudulent information. [...] Yesterday afternoon, the Jiefang Road police station, the reporter saw the portable pseudo-base is made up of two batteries, a set-top box the size of the antenna box and a chassis, as well as a pocket computer composed together at most 5 kg.
(via t byfield and Danny O'Brien)via:mala via:tbyfield privacy scams phishing sms gsm stingray base-stations mobile china
In search of performance - how we shaved 200ms off every POST request — GoCardless Blog
tl;dr: don't use Ruby's Net::HTTP and/or HAProxy prior to 1.4.19
http ruby tcp nagle performance rtt networking haproxy ack curl
Non-Celiac Gluten Sensitivity May Not Exist
The data clearly indicated that a nocebo effect, the same reaction that prompts some people to get sick from wind turbines and wireless internet, was at work here. Patients reported gastrointestinal distress without any apparent physical cause. Gluten wasn't the culprit; the cause was likely psychological. Participants expected the diets to make them sick, and so they did.
gluten placebo nocebo food science health diet gluten-free fodmaps
-
Some nice real-world experimentation around large-scale data processing in differential dataflow:
If you wanted to do an iterative graph computation like PageRank, it would literally be faster to sort the edges from scratch each and every iteration, than to use unsorted edges. If you want to do graph computation, please sort your edges. Actually, you know what: if you want to do any big data computation, please sort your records. Stop talking sass about how Hadoop sorts things it doesn't need to, read some papers, run some tests, and then sort your damned data. Or at least run faster than me when I sort your data for you.
algorithms graphs coding data-processing big-data differential-dataflow radix-sort sorting x-stream counting-sort pagerank
Docker image creation, tagging and traceability in Shippable
this is starting to look quite impressive as a well-integrated Docker-meets-CI model; Shippable is basing its builds off Docker baselines and is automatically cutting Docker images of the post-CI stage. Must take another look
Analysis of PS4's security and the state of hacking
FreeBSD jails and Return-Oriented Programming:
Think of [Return-Oriented Programming] as writing a new chapter to a book, using only words that have appeared at the end of sentences in the previous chapters.
ps4 freebsd jails security exploits hacking sony rop return-oriented-programming
10 Lesser-Known Cocktails You Should Be Drinking
like the sound of some of these
My wife found my email in the Ashley Madison database
On misdirected emails and the potential side-effects:
The reasons why these people give out my email instead of one that they can access have always been a bit mysterious to me. It's one thing to save yourself some spam by using a throwaway address. But why use someone else's for correspondence you actually want to receive? The closest I've come to a working theory is that a lot of them, having been slow off the mark to obtain their own gmail, have addresses like eratliff75@gmail.com. Either they believe they can leave off the numbers and receive the messages anyway, or they often simply forget. That or the E. Ratliffs of the world just view eratliff@gmail.com as some kind of shared resource.
How to Make Raspberry-Thyme Shrub
looks tasty/non-tricky
How gaming terminology is part of modern mainstream Chinese slang
A few years ago, my mom called to ask for my advice on webcams. She explained (in the English-peppered Chinese that's the official language of our Chinese-American household) that some of her friends had started sharing videos of themselves singing karaoke. She thought she could do better. "?????PK??," she remarked: "I want to PK them a little."
buildfarm_deployment/cleanup_docker_images.py
Cleanup old/obsolete Docker images in a repo.
-
Chronos (the Mesos distributed scheduler) comes out looking pretty crappy here
aphyr mesos chronos cron scheduling outages ops jepsen testing partitions cap
Kubernetes and AWS VPC Peering – Ben Straub
the perils of overloading 10/8
How your entire financial life will be stored in a new 'digital vault' - Telegraph
In a move to make it easier to open bank accounts and Isas, people will be asked to share all of their accounts, tax records and personal details with a central service. To check someone's identity, a company would then ask potential customers a series of questions and check the answers against the information in the vault. The checks would replace the current system in which new customers must send by post copies of their passports, cross-signed by a friend, along with bank statements and utility bills.
hahahaha NO FUCKING WAY.bills banking uk tax privacy digital-vault accounts authentication identity-theft bad-ideas
-
_FBGraphQLConnectionStorePersistentPageLoaderOperationDelegate-Protocol.h _FBReactionAcornSportsContentSettingsSetShouldNotPushNotificationsMutationCall.h FBBoostedComponentCreateInputDataCreativeObjectStorySpecLinkDataCallToActionValue.h FBEventUpdateNotificationSubscriptionLevelMutationOptimisticPayloadFactoryProtocol-Protocol.h
I just threw up a little. See also https://www.facebook.com/notes/facebook-engineering/under-the-hood-dalvik-patch-for-facebook-for-android/10151345597798920 , in which the FB Android devs happily reveal that they hot-patch the Dalvik VM at runtime to work around a limit -- rather than refactoring their app.facebook horrors coding ios android dalvik hot-patching apps
-
I can't believe this is the state of food blogging in the UK and Ireland. full-on payola for reviews. See also @damienmulley's excellent rant on the subject in this country: https://twitter.com/damienmulley/status/633353368757497858 -- there's even rate cards for positive review tweets/posts/facebook updates etc.
food blogging restaurants uk bakeries reviews payola blogger-blackmail pr
The reusable holdout: Preserving validity in adaptive data analysis
Useful stats hack from Google: "We show how to safely reuse a holdout data set many times to validate the results of adaptively chosen analyses."
statistics google reusable-holdout training ml machine-learning data-analysis holdout corpus sampling
Recommender Systems (Machine Learning Summer School 2014 @ CMU)
Extremely authoritative slide deck on building a recommendation system, from Xavier Amatriain, Research/Engineering Manager at Netflix
netflix recommendations recommenders ml machine-learning cmu clustering algorithms
-
our full-featured, high performance, scalable web server designed to compete with the likes of nginx. It has been built from the ground-up with no external library dependencies entirely in x86_64 assembly language, and is the result of many years' experience with high volume web environments. In addition to all of the common things you'd expect a modern web server to do, we also include assembly language function hooks ready-made to facilitate Rapid Web Application Server (in Assembler) development.
assembly http performance https ssl x86_64 web ops rwasa tls
The world beyond batch: Streaming 101 - O'Reilly Media
To summarize, in this post I've: Clarified terminology, specifically narrowing the definition of "streaming" to apply to execution engines only, while using more descriptive terms like unbounded data and approximate/speculative results for distinct concepts often categorized under the "streaming" umbrella. Assessed the relative capabilities of well-designed batch and streaming systems, positing that streaming is in fact a strict superset of batch, and that notions like the Lambda Architecture, which are predicated on streaming being inferior to batch, are destined for retirement as streaming systems mature. Proposed two high-level concepts necessary for streaming systems to both catch up to and ultimately surpass batch, those being correctness and tools for reasoning about time, respectively. Established the important differences between event time and processing time, characterized the difficulties those differences impose when analyzing data in the context of when they occurred, and proposed a shift in approach away from notions of completeness and toward simply adapting to changes in data over time. Looked at the major data processing approaches in common use today for bounded and unbounded data, via both batch and streaming engines, roughly categorizing the unbounded approaches into: time-agnostic, approximation, windowing by processing time, and windowing by event time.
streaming batch big-data lambda-architecture dataflow event-processing cep millwheel data data-processing
What the hell is going on with SoundCloud?
tl;dr: major labels.
Despite having revenue coming in from ads and subscriptions, SoundCloud still relies on outside investment. While the company received $150 million in a funding round at the end of last year, it pales next to the reported $526 million Spotify gained in June, and if one report is to be believed, SoundCloud is running very low on cash. Furthermore, sources suggest that potential investors are waiting to see what happens with Sony and Universal before ploughing in more money. With the high sums reported to be involved, it's a stalemate that could potentially break the company whether it decides to pay or not.
soundcloud music mp3 copyright sony universal spotify funding startups
GSMem: Data Exfiltration from Air-Gapped Computers over GSM Frequencies
Holy shit.
Air-gapped networks are isolated, separated both logically and physically from public networks. Although the feasibility of invading such systems has been demonstrated in recent years, exfiltration of data from air-gapped networks is still a challenging task. In this paper we present GSMem, a malware that can exfiltrate data through an air-gap over cellular frequencies. Rogue software on an infected target computer modulates and transmits electromagnetic signals at cellular frequencies by invoking specific memory-related instructions and utilizing the multichannel memory architecture to amplify the transmission. Furthermore, we show that the transmitted signals can be received and demodulated by a rootkit placed in the baseband firmware of a nearby cellular phone.
gsmem gsm exfiltration air-gaps memory radio mobile-phones security papers
An Amazonian's response to "Inside Amazon: Wrestling Big Ideas in a Bruising Workplace" — Medium
excellent response to the NYT hatchet job
-
This is great. Featuring Mount Buggery:
There were no tracks of any sort until they reached Mt Howitt and Stewart, perhaps not quite as fit as he could have been, was finding the going tough after the descent from Mt Speculation. Faced with the prospect of yet another laborious climb he exploded with the words 'What another bugger! I'll call this mountain Mt Buggery.'
and Mount Arsehole:"We always called it Mt Arsehole... Then they came along with all their fancy bloody maps and ideas. Changed it to Mt Arthur. Christ knows why. Bastard of a place anyway!"
swearing australia mount-buggery mount-arsehole nsw victoria places history names mountains
minimaxir/big-list-of-naughty-strings
Late to this one -- a nice list of bad input (Unicode zero-width spaces, etc) for testing
Preventing Dependency Chain Attacks in Maven
using a whitelist of allowed dependency JARs and their SHAs
security whitelisting dependencies coding jar maven java jvm
-
This is excellent -- I wish more companies took this attitude. Applause for Travis CI.
after a couple of weeks of research, we made a decision to offer our expectant mothers AND fathers: 2 weeks before the due date paid at 100% (optional, but recommended); 20 weeks for normal births paid at 100%; 24 weeks for births with complications paid at 100%; Flexible working hours after the 20/24 weeks are complete (part-time arrangements can be made); Your job will be here for you when you return. When we relayed this information to the two US employees, one became a little teary because her last employer (a much bigger and older company), didn't offer anything. This being her second child, it was a huge relief to know she was going to have paid time off with flexibility upon return. While it was a great reaction, it shouldn't happen this way. If you value your employees, you should value their need for time away. At the same time, if you want to hire someone, whether or not they are already pregnant should be irrelevant.
Well exceeding even the Irish maternity leave entitlements, since it covers fathers too. And this is a startup!travisci startups work life family kids paternity-leave maternity-leave
Improving The Weather On Twitter
lovely open-source dataviz improvement for near-term historical rainfall-radar images
dataviz weather rain rainfall radar nws twitter bots graphics ui
Somewhere Over the Rainbow: How to Make Effective Use of Colors in Meteorological Visualizations
Linked from the "Improving the Weather On Twitter" post -- choosing the "best" colour scheme for meteorological visualization. Great dataviz resource post
dataviz colour color meteorological weather nws papers rgb hcl
Reddit comments from a nuclear-power expert
Reddit user "Hiddencamper" is a senior nuclear reactor operator in the US, and regularly posts very knowledgeable comments about reactor operations, safety procedures, and other details. It's fascinating (via Maciej)
via:maciej nuclear-power nuclear atomic power energy safety procedures operations history chernobyl scram
Amazon EC2 2015 Benchmark: Testing Speeds Between AWS EC2 and S3 Regions
Here we are again, a year later, and still no bloody percentiles! Just amateurish averaging. This is not how you measure anything, ffs. Still, better than nothing I suppose
background doc on the Jeep hack
"Remote Exploitation of an Unaltered Passenger Vehicle", by Dr. Charlie Miller (cmiller@openrce.org) and Chris Valasek (cvalasek@gmail.com). QNX, unauthenticated D-Bus, etc.
'Since a vehicle can scan for other vulnerable vehicles and the exploit doesn't require any user interaction, it would be possible to write a worm. This worm would scan for vulnerable vehicles, exploit them with their payload which would scan for other vulnerable vehicles, etc. This is really interesting and scary. Please don't do this. Please.'
Care.data and access to UK health records: patient privacy and public trust
'In 2013, the United Kingdom launched care.data, an NHS England initiative to combine patient records, stored in the machines of general practitioners (GPs), with information from social services and hospitals to make one centralized data archive. One aim of the initiative is to gain a picture of the care being delivered between different parts of the healthcare system and thus identify what is working in health care delivery, and what areas need greater attention and resources. This case study analyzes the complications around the launch of care.data. It explains the historical context of the program and the controversies that emerged in the course of the rollout. It explores problems in management and communications around the centralization effort, competing views on the safety of "anonymous" and "pseudonymous" health data, and the conflicting legal duties imposed on GPs with the introduction of the 2012 Health and Social Care Act. This paper also explores the power struggles in the battle over care.data and outlines the tensions among various stakeholders, including patients, GPs, the Health and Social Care Information Centre (HSCIC), the government, privacy experts and data purchasers. The predominant public policy question that emerges from this review centers on how best to utilize technological advances and simultaneously strike a balance between the many competing interests around health and personal privacy.'
care.data privacy healthcare uk nhs trust anonymity anonymization gps medicine
-
coining a term for the awful buyer's experience on sites like car-hire or air-travel websites
hate-selling conversion marking upselling travel web consumer
How Irish Navy's expertise saved 367 from 30-second sinking in Mediterranean
War-game exercises saved the day:
As the Ribs made their assessment of the situation and began reassuring those on board that help was at hand, the hopelessly overloaded vessel suddenly listed and sank. The sinking took just over 30 seconds. In those 30 seconds, the Captain of the LE Niamh took a number of instant command decisions that saved hundreds of lives. Most of the refugees cannot swim. Their life expectancy in the water would be measured in seconds. The crew of the Ribs immediately began throwing orange lifejackets into the water – encouraging the now frenzied and milling survivors to cling to them. Individuals, then groups clung to the lifejackets – and one another – as the Ribs rallied around trying to keep the floating human mass from dispersal into wider waters and almost certain death. In the meantime, the commander of the LE Niamh managed to manoeuvre close in to the survivors where spare life-rafts were launched into the water. These 25-man inflatable life-rafts were specifically ordered and kept on board the LE Niamh following a "war-gaming" exercise, where the officers and crew envisaged such a nightmare scenario. Had this forward planning not taken place – there would have been no such extra inflatable lifeboats on board.
war-gaming planning navy ireland mediterranean sea boats refugees migration drowning liferafts
-
A well-maintained list with a potted description of each one (via HN)
Advantages of Monolithic Version Control
another Dan Luu post -- good summary of the monorepo's upside
monorepo git mercurial versioning source-control coding dependencies
"A Review Of Criticality Accidents, 2000 Revision"
Authoritative report from LANL on accidents involving runaway nuclear reactions over the years from 1945 to 1999, around the world. Illuminating example of how incident post-mortems are handled in other industries, and (of course) fascinating in its own right
criticality nuclear safety atomic lanl post-mortems postmortems fission
-
but, crucially, with an Andy-Baio-approved archival process. Nicely done -- this is a good example of how to do it
-
Netflix' official test video -- contains various scenarios which exercise frequent tricky edge cases in video compression and playback; A/V sync, shades of black, running water, etc.
How to get your water tested for lead in Dublin
Ossian has written up this very informative post:
Irish Water is writing to thousands of people living in Dublin this week to warn them that their water is supplied through lead pipes. Irish Water says that most people receiving these letters have a level of lead in their water which is above safe limits. So, if you get one of these letters how do you get your water tested? Irish Water is refusing to supply test kits or to test everyone's water who asks. However the HSE's Public Analyst Lab has told me that they will test water for lead for a fee of €10.
ossian-smyth dun-laoghaire dublin drinking-water water lead green hse irish-water health
-
Good advice on production-quality, decent-scale usage of Kinesis in Java with the official library: batching, retries, partial failures, backoff, and monitoring. (Also, jaysus, the AWS Cloudwatch API is awful, looking at this!)
kpl aws kinesis tips java batching streaming production cloudwatch monitoring coding
IrishCycle.com on the Irish Times' terrible victim-blaming anti-cycling op-ed
Even if The Irish Times wants to deny that it has engaged in victim blaming at a high level, it has also clearly errored in fact in a very significant way. It would be more forgiving if this was an isolated editorial. But it's after two days of wrong or misleading coverage, which now seems to be a trend with the newspaper with unbalanced articles or headlines negatively focusing on cycle routes.
-
plenty of stuff out of bounds in Dublin tomoz
India lifts porn ban after widespread outrage - BBC News
After a brief couple of days.
News of the ban caused a furore on Indian social media, with several senior politicians and members of civil society expressing their opposition to the move. The Indian government said that it was merely complying with the Supreme Court order and was committed to the freedom of communication on the Internet. "I reject with contempt the charge that it is a Talibani government, as being said by some of the critics. Our government supports free media, respects communication on social media and has respected freedom of communication always," Mr Prasad told PTI.
india porn filtering isps internet web child-porn censorship
17 of the most important things to ever happen to Irish Twitter
definitive. The David O'Doherty / Not The RTE Guide "your ma" battle is legendary (http://thedailyedge.thejournal.ie/your-ma-david-odoherty-1290482-Jan2014/)
Amazon S3 Introduces New Usability Enhancements
bucket limit increase, and read-after-write consistency in US Standard. About time too! ;)
New study shows Spain's "Google tax" has been a disaster for publishers
A study commissioned by Spanish publishers has found that a new intellectual property law passed in Spain last year, which charges news aggregators like Google for showing snippets and linking to news stories, has done substantial damage to the Spanish news industry. In the short-term, the study found, the law will cost publishers €10 million, or about $10.9 million, which would fall disproportionately on smaller publishers. Consumers would experience a smaller variety of content, and the law "impedes the ability of innovation to enter the market." The study concludes that there's no "theoretical or empirical justification" for the fee.
google news publishing google-tax spain law aggregation snippets economics
Inside the sad, expensive failure of Google+
"It was clear if you looked at the per user metrics, people weren't posting, weren't returning and weren't really engaging with the product," says one former employee. "Six months in, there started to be a feeling that this isn't really working." Some lay the blame on the top-down structure of the Google+ department and a leadership team that viewed success as the only option for the social network. Failures and disappointing data were not widely discussed. "The belief was that we were always just one weird feature away from the thing taking off," says the same employee.
google google+ failures post-mortems business facebook social-media fail bureaucracy vic-gundotra
8,000 sq ft start-up meeting space revealed for Dublin
Neat. this is a good location for post-work user-group meetups and the like (via Oisin)
Introducing Nurse: Auto-Remediation at LinkedIn
Interesting to hear about auto-remediation in prod -- we built a (very targeted) auto-remediation system in Amazon on the Network Monitoring team, but this is much bigger in focus
-
Choco is [FOSS] dedicated to Constraint Programming[2]. It is a Java library written under BSD license. It aims at describing hard combinatorial problems in the form of Constraint Satisfaction Problems and solving them with Constraint Programming techniques. The user models its problem in a declarative way by stating the set of constraints that need to be satisfied in every solution. Then, Choco solves the problem by alternating constraint filtering algorithms with a search mechanism. [...] Choco is among the fastest CP solvers on the market. In 2013 and 2014, Choco has been awarded many medals at the MiniZinc challenge that is the world-wide competition of constraint-programming solvers.
choco constraint-programming solving search combinatorial algorithms
Three Flavours Cornetto trilogy
Shaun Of The Dead, Hot Fuzz, and The World's End are a trilogy. I had no idea! (via David Malone)
Postmortem for July 27 outage of the Manta service - Blog - Joyent
Summary: PostgreSQL's dreaded unpredictable "vacuum" GC
-
YAS3FS (Yet Another S3-backed File System) is a Filesystem in Userspace (FUSE) interface to Amazon S3. It was inspired by s3fs but rewritten from scratch to implement a distributed cache synchronized by Amazon SNS notifications. A web console is provided to easily monitor the nodes of a cluster.
-
RunJOP (Run Just Once Please) is a distributed execution framework to run a command (i.e. a job) only once in a group of servers [built using AWS DynamoDB and S3].
nifty! Distributed cron is pretty easy when you've got Dynamo doing the heavy lifting.dynamodb cron distributed-cron scheduling runjop danilop hacks aws ops
Why Docker is Not Yet Succeeding Widely in Production
Spot-on points which Docker needs to address. It's still production-ready, and _should_ be used there, it just has significant rough edges...
How to Create RSS Feeds for Twitter
The latest hacky workaround to Twitter's API shortcoming
rss-feeds feeds twitter favorites api social-media workaround google-script
Testing without mocking in Scala
mocks are the sound of your code crying out, "please structure me differently!"
+1scala via:jessitron mocks mock-objects testing testability coding
Newegg vs. Patent Trolls: When We Win, You Win
go NewEgg: 'Newegg went against a company that claimed its patent covered SSL and RC4 encryption, a common encryption system used by many retailers and websites. This particular patent troll has gone against over 100 other companies, and brought in $45 million in settlements before going after Newegg. We won.'
-
A lovely eulogy for Nóirín Plunkett, from Rich Bowen. RIP Nóirín :(
A Visual Introduction to Machine Learning
beautiful visualisation of a decision tree
Taming Complexity with Reversibility
This is a great post from Kent Beck, putting a lot of recent deployment/rollout patterns in a clear context -- that of supporting "reversibility":
Development servers. Each engineer has their own copy of the entire site. Engineers can make a change, see the consequences, and reverse the change in seconds without affecting anyone else. Code review. Engineers can propose a change, get feedback, and improve or abandon it in minutes or hours, all before affecting any people using Facebook. Internal usage. Engineers can make a change, get feedback from thousands of employees using the change, and roll it back in an hour. Staged rollout. We can begin deploying a change to a billion people and, if the metrics tank, take it back before problems affect most people using Facebook. Dynamic configuration. If an engineer has planned for it in the code, we can turn off an offending feature in production in seconds. Alternatively, we can dial features up and down in tiny increments (i.e. only 0.1% of people see the feature) to discover and avoid non-linear effects. Correlation. Our correlation tools let us easily see the unexpected consequences of features so we know to turn them off even when those consequences aren't obvious. IRC. We can roll out features potentially affecting our ability to communicate internally via Facebook because we have uncorrelated communication channels like IRC and phones. Right hand side units. We can add a little bit of functionality to the website and turn it on and off in seconds, all without interfering with people's primary interaction with NewsFeed. Shadow production. We can experiment with new services under real load, from a tiny trickle to the whole flood, without affecting production. Frequent pushes. Reversing some changes require a code change. On the website we never more than eight hours from the next schedule code push (minutes if a fix is urgent and you are willing to compensate Release Engineering). The time frame for code reversibility on the mobile applications is longer, but the downward trend is clear from six weeks to four to (currently) two. Data-informed decisions. (Thanks to Dave Cleal) Data-informed decisions are inherently reversible (with the exceptions noted below). "We expect this feature to affect this metric. If it doesn't, it's gone." Advance countries. We can roll a feature out to a whole country, generate accurate feedback, and roll it back without affecting most of the people using Facebook. Soft launches. When we roll out a feature or application with a minimum of fanfare it can be pulled back with a minimum of public attention. Double write/bulk migrate/double read. Even as fundamental a decision as storage format is reversible if we follow this format: start writing all new data to the new data store, migrate all the old data, then start reading from the new data store in parallel with the old.
We do a bunch of these in work, and the rest are on the to-do list. +1 to these!software deployment complexity systems facebook reversibility dark-releases releases ops cd migration
Benchmarking GitHub Enterprise - GitHub Engineering
Walkthrough of debugging connection timeouts in a load test. Nice graphs (using matplotlib)
github listen-backlog tcp debugging timeouts load-testing benchmarking testing ops linux
How .uk came to be (and why it's not .gb)
WB: By the late 80s the IANA [the Internet Assigned Numbers Authority, set up in 1988 to manage global IP address allocations] was trying to get all those countries that were trying to join the internet to use the ISO 3166 standard for country codes. It was used for all sorts of things?—?you see it on cars, "GB" for the UK. [...] At that point, we're faced with a problem that Jon Postel would like to have changed it to .gb to be consistent with the rest of the world. Whereas .uk had already been established, with a few tens of thousands of domain names with .uk on them. I remember chairing one of the JANET net workshops that were held every year, and the Northern Irish were adamant that they were part of the UK?—?so the consensus was, we'd try and keep .uk, we'd park .gb and not use it. PK: I didn't particularly want to change to .gb because I was responsible for Northern Ireland as well. And what's more, there was a certain question as to whether a research group in the US should be allowed to tell the British what to do. So this argy-bargy continued for a little while and, in the meantime, one of my clients was the Ministry of Defence, and they decided they couldn't wait this long, and they decided I was going to lose the battle, and so bits of MOD went over to .gb?—?I didn't care, as I was running .gb and .uk in any case.
dot-uk history internet dot-gb britain uk northern-ireland ireland janet
That time the Internet sent a SWAT team to my mom's house - Boing Boing
The solution is for social media sites and the police to take threats or jokes about swatting, doxxing, and organized crime seriously. Tweeting about buying a gun and shooting up a school would be taken seriously, and so should the threat of raping, doxxing, swatting or killing someone. Privacy issues and online harassment are directly linked, and online harassment isn't going anywhere. My fear is that, in reaction to online harassment, laws will be passed that will break down our civil freedoms and rights online, and that more surveillance will be sold to users under the guise of safety. More surveillance, however, would not have helped me or my mother. A platform that takes harassment and threats seriously instead of treating them like jokes would have.
twitter gamergate 4chan 8chan privacy doxxing swatting harrassment threats social-media facebook law feminism
Why Google's Deep Dream Is Future Kitsch
Deep Dream estranges us from our fears, perhaps, but it doesn't make them go away. It's easy to discuss Deep Dream as an independent creature, a foreign intelligence that we interact with for fun. Yet like all kitsch, it comes straight back to its creators.
It's Not Climate Change?—?It's Everything Change
now this is a Long Read. the inimitable Margaret Atwood on climate change, beautifully illustrated
climate climate-change margaret-atwood long-reads change life earth green future
In Praise of the AK-47 — Dear Design Student — Medium
While someone can certainly make the case that an AK-47, or any other kind of gun or rifle is designed, nothing whose primary purpose is to take away life can be said to be designed well. And that attempting to separate an object from its function in order to appreciate it for purely aesthetic reasons, or to be impressed by its minimal elegance, is a coward's way of justifying the death they've designed into the word, and the money with which they're lining their pockets.
-
turns out Ruby has a good set of random-text-generation gems on offer
The Titanium Gambit | History | Air & Space Magazine
Amazing story of 1960s detente via Maciej: 'During the Cold War, Boeing execs got a strange call from the State Department: Would you guys mind trading secrets with the Russians?'
via:maciej titanium history cold-war detente ussr usa boeing russia aerospace
I've seen more than my fair share of abuse online, but Lorraine Higgins' bill isn't the answer
Tom Murphy:
This bill prioritises other peoples' "alarm or distress" over your communications not just TO them but also ABOUT them. Don't like what Joan Burton is doing with the water charges? Want to write something on independent media about what you think of that? Better not alarm or distress or harm her! This is the core of my issue with the bill. It's not just that almost all the agreeable parts of it are already covered by other laws. It's not just that it's utterly unenforceable with our current justice system. It's not just that it's so vague and fluffy. It's that it's so ill-defined and over-reaching that its interpretation will inevitably have to be left to judges. Leaving anything to judges is a bad idea in general. This overly broad and poorly worded bill is a god-send to people who like to bully others into silence. Ironic that eh?!
-
Lambdas in Java 8 introduce some unpredictable performance implications, due to reliance on escape analysis to eliminate object allocation on every lambda invocation. Peter Lawrey has some details
lambdas java-8 java performance low-latency optimization peter-lawrey coding escape-analysis
Mikhail Panchenko's thoughts on the July 2015 CircleCI outage
an excellent followup operational post on CircleCI's "database is not a queue" outage
database-is-not-a-queue mysql sql databases ops outages postmortems
Men who harass women online are quite literally losers, new study finds
(1) players are anonymous, and the possibility of "policing individual behavior is almost impossible"; (2) they only encounter each other a few times in passing — it's very possible to hurl an expletive at another player, and never "see" him or her again; and (3) finally, and perhaps predictably, the sex-ratio of players is biased pretty heavily toward men. (A 2014 survey of gender ratios on Reddit found that r/halo was over 95 percent male.) [....] In each of these environments, Kasumovic suggests, a recent influx of female participants has disrupted a pre-existing social hierarchy. That's okay for the guys at the top — but for the guys at the bottom, who stand to lose more status, that's very threatening. (It's also in keeping with the evolutionary framework on anti-lady hostility, which suggests sexism is a kind of Neanderthal defense mechanism for low-status, non-dominant men trying to maintain a shaky grip on their particular cave's supply of women.) "As men often rely on aggression to maintain their dominant social status," Kasumovic writes, "the increase in hostility towards a woman by lower-status males may be an attempt to disregard a female's performance and suppress her disturbance on the hierarchy to retain their social rank."
losers sexism mysogyny women halo gaming gamergate 4chan abuse harrassment papers bullying social-status
The old suburban office park is the new American ghost town - The Washington Post
Most analyses of the market indicate that office parks simply aren't as appealing or profitable as they were in the 20th century and that Americans just aren't as keen to cloister themselves in workspaces that are reachable only by car.
cbd cities work life office-parks commuting america history workplaces
HACKERS REMOTELY KILL A JEEP ON THE HIGHWAY—WITH ME IN IT
Jaysus, this is terrifying.
Miller and Valasek's full arsenal includes functions that at lower speeds fully kill the engine, abruptly engage the brakes, or disable them altogether. The most disturbing maneuver came when they cut the Jeep's brakes, leaving me frantically pumping the pedal as the 2-ton SUV slid uncontrollably into a ditch.
Avoid any car which supports this staggeringly-badly-conceived Uconnect feature:All of this is possible only because Chrysler, like practically all carmakers, is doing its best to turn the modern automobile into a smartphone. Uconnect, an Internet-connected computer feature in hundreds of thousands of Fiat Chrysler cars, SUVs, and trucks, controls the vehicle's entertainment and navigation, enables phone calls, and even offers a Wi-Fi hot spot.
:facepalm: Also, Chrysler's response sucks: "Chrysler's patch must be manually implemented via a USB stick or by a dealership mechanic."hacking security cars driving safety brakes jeeps chrysler fiat uconnect can-bus can
RFC 3339: Date and Time on the Internet: Timestamps
the RFC take on ISO-8601. I need to update my mental bookmarks to start referring to this instead
"Customer data is a liability, not an asset."
Great turn of phrase from Matthew Green (@matthew_d_green). Emin Gün Sirer adds some detail: "well, an asset with bounded value, and an unbounded liability"
data privacy data-protection ashleymadison hacks security liability
Deep Dive Into Docker Storage Drivers
good detail in this presentation
docker overlayfs aufs btrfs filesystems ops linux containers
-
oh look, Google has a flight search engine! I had no idea
-
TorrentFreak has the story of a UK-producer and songwriter named Lee Adams who took part in an official remix competition of boy band One Direction's music, put on by the band and its label, Sony Music. The stems for remixing were released on Soundcloud. The rules of the contest required entrants to upload their remixes on Soundcloud... and that's exactly what Adams did. And yet those works still got taken down via copyright claims from Sony Music as infringing.
sony soundcloud anti-piracy automation piracy stems remixing one-direction lee-adams
WereBank | Were Bank Energy for the People
The Freeman-On-The-Land movement is starting a bank. lols guaranteed
freemen funny werebank banking money on-my-oath maritime-law
Angela Merkel told a sobbing girl she couldn't save her from deportation. It was a lie. - Vox
Argentina has, as a matter of constitutional law, effectively open borders. There are no caps or quotas or lottery systems. You can move there legally if you have an employer or family member to sponsor you. That's all you need. If you don't have a sponsor, and make your way in illegally, you're recognized as an "irregular migrant." Discrimination against irregular migrants in health care or education is illegal, and deportation in noncriminal cases is exceptionally rare. Large-scale amnesties are the norm. Obviously Argentina is not nearly as rich as Germany or the US or the UK. But it's considerably richer than three of its neighbors (Bolivia, Paraguay, and Brazil). And yet it doesn't try hard to keep their residents out. It welcomes them — as it should. "One could have expected catastrophe—an uncontrollable flow of poorer immigrants streaming into the country coupled with angry public backlash," Elizabeth Slater writes in the World Policy Journal. "That hasn't happened." Angela Merkel clearly expects catastrophe if she lets people like this weeping young Palestinian girl stay in Germany. That catastrophe is simply a myth; it wouldn't happen. What would happen is that Germany's economy would grow, its culture would grow richer, and that girl and more like her could see their lives improve immeasurably.
argentina immigration angela-merkel germany eu migrants deportation economics
-
'A programming language based on the one liners of Arnold Schwarzenegger'. Presenting hello.arnoldc: IT'S SHOWTIME TALK TO THE HAND "hello world" YOU HAVE BEEN TERMINATED (via Robert Walsh)
via:rjwalsh c arnold-schwarzenegger one-liners funny coding silly languages
A simple guide to 9-patch for Android UI
This is a nifty hack. TIL! '9-patch uses png transparency to do an advanced form of 9-slice or scale9. The guides are straight, 1-pixel black lines drawn on the edge of your image that define the scaling and fill of your image. By naming your image file name.9.png, Android will recognize the 9.png format and use the black guides to scale and fill your bitmaps.'
android design 9-patch scaling images bitmaps scale9 9-slice ui graphics
Government forum to discuss increasing use of personal data
Mr Murphy said it was the Government's objective for Ireland to be a leader on data protection and data-related issues. The members of the forum include Data Protection Commissioner Helen Dixon, John Barron, chief technology officer with the Revenue Commissioners, Seamus Carroll, head of civil law reform division at the Department of Justice and Tim Duggan, assistant secretary with the Department of Social Protection. Gary Davis, director of privacy and law enforcement requests with Apple, is also on the forum. Mr Davis is a former deputy data protection commissioner in Ireland. There are also representatives from Google, Twitter, LinkedIn and Facebook, from the IDA, the Law Society and the National Statistics Board. Chair of Digital Rights Ireland Dr TJ McIntyre and Dr Eoin O'Dell, associate professor, School of Law, Trinity College Dublin are also on the voluntary forum.
-
From DataDog. See also "How to collect NGINX metrics": https://www.datadoghq.com/blog/how-to-collect-nginx-metrics/
From Zero to Docker: Migrating to the Whale
nicely detailed writeup of how New Relic are dockerizing
Docker with OverlayFS first impressions
a brief howto
-
a new sketch algorithm from Baron Schwartz and Preetam Jinka of VividCortex; similar to Count-Min but with last-seen timestamp instead of frequency.
sketch algorithms estimation approximation sampling streams big-data
-
The UK Ordnance Survey's "open data' free product, free for all uses:
Code-Point Open is FREE to view, download and use for commercial, educational and personal purposes.
(via Antoin)via:antoin postcodes mapping open-data ordnance-survey uk gb royal-mail maps
Apple now biases towards IPv6 with a 25ms delay on connections
Interestingly, they claim that IPv6 tends to be more reliable and has lower latency now:
Based on our testing, this makes our Happy Eyeballs implementation go from roughly 50/50 IPv4/IPv6 in iOS 8 and Yosemite to ~99% IPv6 in iOS 9 and El Capitan betas. While our previous implementation from four years ago was designed to select the connection with lowest latency no matter what, we agree that the Internet has changed since then and reports indicate that biasing towards IPv6 is now beneficial for our customers: IPv6 is now mainstream instead of being an exception, there are less broken IPv6 tunnels, IPv4 carrier-grade NATs are increasing in numbers, and throughput may even be better on average over IPv6.
apple ipv6 ip tcp networking internet happy-eyeballs ios osx
-
lest we forget -- this is a 2014-era writeup of OpenPostcode (open), Loc8 and GoCode (proprietary) as alternative options to the Eircode system
eircode openpostcode loc8 gocode ireland geocoding mapping location history open-data
-
handy step-by-step clickthrough guide
Outlier Detection at Netflix | Hacker News
Excellent HN thread re automated anomaly detection in production, Q&A with the dev team
machine-learning ml remediation anomaly-detection netflix ops time-series clustering
-
A new HTTP client library for Android and Java, with a lot of nice features:
HTTP/2 and SPDY support allows all requests to the same host to share a socket. Connection pooling reduces request latency (if SPDY isn't available). Transparent GZIP shrinks download sizes. Response caching avoids the network completely for repeat requests. OkHttp perseveres when the network is troublesome: it will silently recover from common connection problems. If your service has multiple IP addresses OkHttp will attempt alternate addresses if the first connect fails. This is necessary for IPv4+IPv6 and for services hosted in redundant data centers. OkHttp initiates new connections with modern TLS features (SNI, ALPN), and falls back to TLS 1.0 if the handshake fails. Using OkHttp is easy. Its 2.0 API is designed with fluent builders and immutability. It supports both synchronous blocking calls and async calls with callbacks.
android http java libraries okhttp http2 spdy microservices jdk
-
via Ossian.
via:smytho tech-specs specs eircode addresses geocoding ireland mapping
AWS Best Practices for DDoS Resiliency [pdf]
Reasonably solid white paper
Self-driving cars drive like your grandma
'Honestly, I don't think it will take long for other drivers to realize that self-driving cars are "easy targets" in traffic.' -- also, an insurance expert suggests that self-driving cars won't increase premiums
-
NZ's HDC Act gets the EFF thumbs-down
jgc on Cloudflare's log pipeline
Cloudflare are running a 40-machine, 50TB Kafka cluster, ingesting at 15 Gbps, for log processing. Also: Go producers/consumers, capnproto as wire format, and CitusDB/Postgres to store rolled-up analytics output. Also using Space Saver (top-k) and HLL (counting) estimation algorithms.
logs cloudflare kafka go capnproto architecture citusdb postgres analytics streaming
-
a command line tool for JVM diagnostic troubleshooting and profiling.
-
'Simplistic interactive filtering tool' -- live incremental-search filtering in a terminal window
cli shell terminal tools go peco interactive incremental-search search ui unix
Google Cloud Platform announces new Container Registry
Yay. Sensible Docker registry pricing at last. Given the high prices, rough edges and slow performance of the other registry offerings, I'm quite happy to see this.
Google Container Registry helps make it easy for you to store your container images in a private and encrypted registry, built on Cloud Platform. Pricing for storing images in Container Registry is simple: you only pay Google Cloud Storage costs. Pushing images is free, and pulling Docker images within a Google Cloud Platform region is free (Cloud Storage egress cost when outside of a region). Container Registry is now ready for production use: * Encrypted and Authenticated - Your container images are encrypted at rest, and access is authenticated using Cloud Platform OAuth and transmitted over SSL * Fast - Container Registry is fast and can handle the demands of your application, because it is built on Cloud Storage and Cloud Networking. * Simple - If you're using Docker, just tag your image with a gcr.io tag and push it to the registry to get started. Manage your images in the Google Developers Console. * Local - If your cluster runs in Asia or Europe, you can now store your images in ASIA or EU specific repositories using asia.gcr.io and eu.gcr.io tags.
docker registry google gcp containers cloud-storage ops deployment
Docker at Shopify: From This-Looks-Fun to Production
Pragmatic evolution story, adding Docker as a packaging/deploy format for an existing production Capistrano/Rails fleet
-
Hystrix-style Circuit Breakers and Bulkheads for Ruby/Rails, from Shopify
circuit-breaker bulkhead patterns architecture microservices shopify rails ruby networking reliability fallback fail-fast
Brubeck, a statsd-compatible metrics aggregator - GitHub Engineering
GitHub's statsd replacement in C
Patrick Shuff - Building A Billion User Load Balancer - SCALE 13x - YouTube
'Want to learn how Facebook scales their load balancing infrastructure to support more than 1.3 billion users? We will be revealing the technologies and methods we use to route and balance Facebook's traffic. The Traffic team at Facebook has built several systems for managing and balancing our site traffic, including both a DNS load balancer and a software load balancer capable of handling several protocols. This talk will focus on these technologies and how they have helped improve user performance, manage capacity, and increase reliability.' Can't find the standalone slides, unfortunately.
facebook video talks lbs load-balancing http https scalability scale linux
-
a good collection of coding fonts (via Tony Finch)
-
Finagle Futures ported to C++11
-
"Shoggoth ovipositors":
So then they reach inside to one of the layers and spin the knob randomly to fuck it up. Lower layers are edges and curves. Higher layers are faces, eyes and shoggoth ovipositors. [....] But the best part is not when they just glitch an image -- which is a fun kind of embossing at one end, and the "extra eyes" filter at the other -- but is when they take a net trained on some particular set of objects and feed it static, then zoom in, and feed the output back in repeatedly. That's when you converge upon the platonic ideal of those objects, which -- it turns out -- tend to be Giger nightmare landscapes. Who knew. (I knew.)
This stuff is still boggling my mind. All those doggy faces! That is one dog-obsessed ANN.neural-networks ai jwz funny shoggoths image-recognition hr-giger art inceptionism
Levenshtein automata can be simple and fast
Nice algorithm for fuzzy text search with a limited Levenshtein edit distance using a DFA
dfa algorithms levenshtein text edit-distance fuzzy-search search python
Discretized Streams: Fault Tolerant Stream Computing at Scale
The paper describing the innards of Spark Streaming and its RDD-based recomputation algorithm:
we use a data structure called Resilient Distributed Datasets (RDDs), which keeps data in memory and can recover it without replication by tracking the lineage graph of operations that were used to build it. With RDDs, we show that we can attain sub-second end-to-end latencies. We believe that this is sufficient for many real-world big data applications, where the timescale of the events tracked (e.g., trends in social media) is much higher.
rdd spark streaming fault-tolerance batch distcomp papers big-data scalability
Improving testing by using real traffic from production
Gor, a very nice-looking tool to log and replay HTTP traffic, specifically designed to "tee" live traffic from production to staging for pre-release testing
gor performance testing http tcp packet-capture tests staging tee
Git team workflows: merge or rebase?
Well-written description of the pros and cons. I'm a rebaser, fwiw. (via Darrell)
via:darrell git merging rebasing history git-log coding workflow dev teams collaboration github
How to receive a million packets per second on Linux
To sum up, if you want a perfect performance you need to: Ensure traffic is distributed evenly across many RX queues and SO_REUSEPORT processes. In practice, the load usually is well distributed as long as there are a large number of connections (or flows). You need to have enough spare CPU capacity to actually pick up the packets from the kernel. To make the things harder, both RX queues and receiver processes should be on a single NUMA node.
linux networking performance cloudflare packets numa so_reuseport sockets udp
Inceptionism: Going Deeper into Neural Networks
This is amazing, and a little scary.
If we choose higher-level layers, which identify more sophisticated features in images, complex features or even whole objects tend to emerge. Again, we just start with an existing image and give it to our neural net. We ask the network: "Whatever you see there, I want more of it!" This creates a feedback loop: if a cloud looks a little bit like a bird, the network will make it look more like a bird. This in turn will make the network recognize the bird even more strongly on the next pass and so forth, until a highly detailed bird appears, seemingly out of nowhere.
An enlightening comment from the G+ thread:This is the most fun we've had in the office in a while. We've even made some of those 'Inceptionistic' art pieces into giant posters. Beyond the eye candy, there is actually something deeply interesting in this line of work: neural networks have a bad reputation for being strange black boxes that that are opaque to inspection. I have never understood those charges: any other model (GMM, SVM, Random Forests) of any sufficient complexity for a real task is completely opaque for very fundamental reasons: their non-linear structure makes it hard to project back the function they represent into their input space and make sense of it. Not so with backprop, as this blog post shows eloquently: you can query the model and ask what it believes it is seeing or 'wants' to see simply by following gradients. This 'guided hallucination' technique is very powerful and the gorgeous visualizations it generates are very evocative of what's really going on in the network.?
art machine-learning algorithm inceptionism research google neural-networks learning dreams feedback graphics
Apple to switch APNS protocol to HTTP/2
This is great news -- the current protocol is a binary, proprietary horrorshow, particularly around error reporting. Available "later this year" in production, and Pushy plan to support it.
Comparing the Defect Reduction Benefits of Code Inspection and Test-Driven Development
tl;dr: Code review trumps TDD alone for finding bugs. (Via Mark Dennehy)
via:markdennehy code-review coding tdd unit-tests testing papers bugs
Evidence-Based Software Engineering
Objective: Our objective is to describe how software engineering might benefit from an evidence-based approach and to identify the potential difficulties associated with the approach. Method: We compared the organisation and technical infrastructure supporting evidence-based medicine (EBM) with the situation in software engineering. We considered the impact that factors peculiar to software engineering (i.e. the skill factor and the lifecycle factor) would have on our ability to practice evidence-based software engineering (EBSE). Results: EBSE promises a number of benefits by encouraging integration of research results with a view to supporting the needs of many different stakeholder groups. However, we do not currently have the infrastructure needed for widespread adoption of EBSE. The skill factor means software engineering experiments are vulnerable to subject and experimenter bias. The lifecycle factor means it is difficult to determine how technologies will behave once deployed. Conclusions: Software engineering would benefit from adopting what it can of the evidence approach provided that it deals with the specific problems that arise from the nature of software engineering.
(via Mark Dennehy)papers toread via:markdennehy software coding ebse evidence-based-medicine medicine research
Amazon offer a WhatsMyIp service as part of AWS
curl -s http://checkip.amazonaws.com/
-
The ruling is terrible through and through. First off, it insists that the comments on the news story were clearly "hate speech" and that, as such, "did not require any linguistic or legal analysis since the remarks were on their face manifestly unlawful." To the court, this means that it's obvious such comments should have been censored straight out. That's troubling for a whole host of reasons at the outset, and highlights the problematic views of expressive freedom in Europe. Even worse, however, the Court then notes that freedom of expression is "interfered with" by this ruling, but it doesn't seem to care -- saying that it is deemed "necessary in a democratic society."
This is going to have massive chilling effects. Terrible ruling from the ECHR.echr freedom via:tjmcintyre law europe eu comments free-speech censorship hate-speech
Shock European court decision: Websites are liable for users' comments | Ars Technica
In the wake of this judgment, the legal situation is complicated. In an e-mail to Ars, T J McIntyre, who is a lecturer in law and Chairman of Digital Rights Ireland, the lead organization that won an important victory against EU data retention in the Court of Justice of the European Union last year, explained where things now stand. "Today's decision doesn't have any direct legal effect. It simply finds that Estonia's laws on site liability aren't incompatible with the ECHR. It doesn't directly require any change in national or EU law. Indirectly, however, it may be influential in further development of the law in a way which undermines freedom of expression. As a decision of the Grand Chamber of the ECHR it will be given weight by other courts and by legislative bodies."
ars-technica delfi free-speech eu echr tj-mcintyre law europe estonia
Google Cloud Platform Blog: A look inside Google's Data Center Networks
We used three key principles in designing our datacenter networks: We arrange our network around a Clos topology, a network configuration where a collection of smaller (cheaper) switches are arranged to provide the properties of a much larger logical switch. We use a centralized software control stack to manage thousands of switches within the data center, making them effectively act as one large fabric. We build our own software and hardware using silicon from vendors, relying less on standard Internet protocols and more on custom protocols tailored to the data center.
Automated Nginx Reverse Proxy for Docker
Nice hack. An automated nginx reverse proxy which regenerates as the Docker containers update
6 Reasons Modern Movie CGI Looks Surprisingly Crappy
Spot on
color-grading teal-and-orange cgi movies film sfx jurassic-world
Cover Story: "Playdate" - The New Yorker
the story behind Chris Ware's lovely Minecraft New Yorker cover
How We Moved Our API From Ruby to Go and Saved Our Sanity
Parse on their ditching-Rails story. I haven't heard a nice thing about Ruby or Rails as an operational, production-quality platform in a long time :(
-
we are introducing Flow Logs for the Amazon Virtual Private Cloud. Once enabled for a particular VPC, VPC subnet, or Elastic Network Interface (ENI), relevant network traffic will be logged to CloudWatch Logs for storage and analysis by your own applications or third-party tools. You can create alarms that will fire if certain types of traffic are detected; you can also create metrics to help you to identify trends and patterns. The information captured includes information about allowed and denied traffic (based on security group and network ACL rules). It also includes source and destination IP addresses, ports, the IANA protocol number, packet and byte counts, a time interval during which the flow was observed, and an action (ACCEPT or REJECT).
ec2 aws vpc logging tracing ops flow-logs network tcpdump packets packet-capture
Tim Hunt "jokes" about women scientists. Or not. (with image, tweets) · deborahblum · Storify
'[Tim Hunt] said that while he meant to be ironic, he did think it was hard to collaborate with women because they are too emotional - that he was trying to be honest about the problems.' So much for the "nasty twitter took my jokes seriously" claims then.
twitter science misogyny women tim-hunt deborah-blum journalism
-
Good post, and hard to disagree.
One of the "features" of systemd is that it allows you to boot a system without needing a shell at all. This seems like such a senseless manoeuvre that I can't help but think of it as a knee-jerk reaction to the perception of Too Much Shell in sysv init scripts. In exactly which universe is it reasonable to assume that you have a running D-Bus service (or kdbus) and a filesystem containing unit files, all the binaries they refer to, all the libraries they link against, and all the configuration files any of them reference, but that you lack that most ubiquitous of UNIX binaries, /bin/sh?
Adrian Colyer reviews the Twitter Heron paper
ouch, really sounds like Storm didn't cut the muster. 'It's hard to imagine something more damaging to Apache Storm than this. Having read it through, I'm left with the impression that the paper might as well have been titled "Why Storm Sucks", which coming from Twitter themselves is quite a statement.' If I was to summarise the lessons learned, it sounds like: backpressure is required; and multi-tenant architectures suck.
storm twitter heron big-data streaming realtime backpressure
Security theatre at Allied Irish Banks
Allied Irish Banks's web and mobile banking portals are ludicrously insecure. Vast numbers of accounts have easily-guessable registration numbers and are thus 'protected' by a level of security that is twice as easy to crack as would be provided by a single password containing only two lowercase letters. A person of malicious intent could easily gain access to hundreds, possibly thousands, of accounts as well as completely overwhelm the branch network by locking an estimated several 100,000s of people out of their online banking. Both AIB and the Irish Financial Services Ombudsman have refused to respond meaningfully to multiple communications each in which these concerns were raised privately.
Leveraging AWS to Build a Scalable Data Pipeline
Nice detailed description of an auto-scaled SQS worker pool
sqs aws ec2 auto-scaling asg worker-pools architecture scalability
China's Spies Hit the Blackmail Jackpot With Data on 4 Million Federal Workers
The Daily Beast is scathing re the OPM hack:
Here's where things start to get scary. Whoever has OPM's records knows an astonishing amount about millions of federal workers, members of the military, and security clearance holders. They can now target those Americans for recruitment or influence. After all, they know their vices, every last one—the gambling habit, the inability to pay bills on time, the spats with former spouses, the taste for something sexual on the side—since all that is recorded in security clearance paperwork. (To get an idea of how detailed this gets, you can see the form, called an SF86, here.) Speaking as a former counterintelligence officer, it really doesn't get much worse than this.
daily-beast sf86 clearance us-government america china cyberwar hacking opm privacy
-
'Easy Skeezy Ruby Date/Time Formatting' -- or indeed anywhere else strftime() is supported
-
'a fully-automated solution to build auto-scaling etcd clusters in AWS'
-
New static analysis goodnews, freshly open-sourced by Facebook:
Facebook Infer uses logic to do reasoning about a program's execution, but reasoning at this scale — for large applications built from millions of lines of source code — is hard. Theoretically, the number of possibilities that need to be checked is more than the number of estimated atoms in the observable universe. Furthermore, at Facebook our code is not a fixed artifact but an evolving system, updated frequently and concurrently by many developers. It is not unusual to see more than a thousand modifications to our mobile code submitted for review in a given day. The requirements on the program analyzer then become even more challenging because we expect a tool to report quickly on these code modifications — in the region of 10 minutes — to fit in with developers' workflow. Coping with this scale and velocity requires advanced mathematical techniques. Facebook Infer uses two such techniques: separation logic and bi-abduction. Separation logic is a theory that allows Facebook Infer's analysis to reason about small, independent parts of the application storage, rather than having to consider the entirety of the memory potentially at every step. That would be a daunting task on modern processors with their large addressable virtual memories. Bi-abduction is a logical inference technique that allows Facebook Infer to discover properties about the behavior of independent parts of the application code. By storing these properties between runs, Facebook Infer needs to analyze only the parts of the software that have changed, reusing the results of its previous analysis where it can. By combining these approaches, our analyzer is able to find complex problems in modifications to an application built from millions of lines of code, in minutes.
(via Bryan O'Sullivan)via:bos infer facebook static-analysis lint code java ios android coding bugs
The Tamborzão Goes to Thailand
This is great. the story of how cheesy funk carioca tune "A Minha Amiga Fran" managed to become "Kawo Kawo" and become a massive hit in Thailand
AV vendors still relying on MD5 to identify malware
oh dear. I can see how this happened -- in many cases they may not still have samples to derive new sums from :(
Google Photos - Can I get out?
what's the export policy for Google's new Photos service? pretty good, it turns out
google export data google-photos photos archive history storage
A higher order estimate of the optimum checkpoint interval for restart dumps
tl;dr:
the bottom line is as follows: If the time it takes to create a dump, ? < M/2 then use ?opt = ?(2?M) – ? Otherwise (it takes longer than M/2 to create a dump), just use ?opt = M.
dumping periodic-tasks scheduling frequency maths optimal interval checkpointing
-
Creating a user with our DSL looks like: registrationAPI.createUser("user"); You might expect this to create a user with the username 'user', but then we'd get conflicts between every test that wanted to call their user 'user' which would prevent tests from running safely against the same deployment of the exchange. Instead, 'user' is just an alias that is only meaningful while this one test is running. The DSL creates a unique username that it uses when talking to the actual system. Typically this is done by adding a postfix so the real username is still reasonably understandable e.g. user-fhoai42lfkf.
Nice approach -- makes sense. -
Orbit Async implements async-await methods in the JVM. It allows programmers to write asynchronous code in a sequential fashion. It was developed by BioWare, a division of Electronic Arts.
Open source, BSD-licensed. Who wrote this amazing, mysterious book satirizing tech startup culture?
very cool
books reading startups silicon-valley mysteries pranks san-francisco
1172401 – Add Amazon root certificates
Well, well -- looks like AWS is about to disrupt PKI, and about time too. If they come up with a Plex-style "provision a cert" API, it'll be revolutionary
Vintage Illustrations for Tolkien's The Hobbit from Around the World | Brain Pickings
including a lovely set from Tove Jansson
tove-jansson art illustration tolkien the-hobbit books via:ianmoore
How Plex is doing HTTPS for all its users
large-scale automated TLS certificate deployment. very impressive and not easy to reproduce, good work Plex! (via Nelson)
via:nelson https ssl tls certificates pki digicert security plex
Tuning Java Garbage Collection for Spark Applications
So much for G1GC being fire-and-forget
-
Airbnb's workflow management system; works off a DAG defined in Python code (ugh). Nice UI though, but I think Pinboard's take is neater
airbnb open-source python workflow jobs cron scheduling batch
A Complete Taxonomy of Internet Chum - The Awl
Introducing the chumbox
-
A high-performance java build tool, from Facebook. Make-like
-
in favour of a proprietary ground-up rewrite called Heron. Reading between the lines it sounds like Storm had problems with latency, reliability, data loss, and supporting back pressure.
analytics architecture twitter storm heron backpressure streaming realtime queueing
-
neat substitute for physical-time clocks in synchronization and ordering in a distributed system, based on Lamport's Logical Clocks and Google's TrueTime. 'HLC captures the causality relationship like LC, and enables easy identification of consistent snapshots in distributed systems. Dually, HLC can be used in lieu of PT clocks since it maintains its logical clock to be always close to the PT clock.'
hlc clocks logical-clocks time synchronization ordering events logs papers algorithms truetime distcomp
-
Increasingly bizarre postal address obfuscation with An Post, the Irish postal service. Example:
I have decided to see what you can post [....] My first experiment was a dice [sic] with one line of the address on each side. An Post delivered two days later. They win this round
Via JG -
'Can do ~1M queries to ~3K public DNS servers within ~3 minutes with just a few threads.' via Trustin Lee. Netty is the business
-
Good series of blog posts on the LMAX trading platform's performance testing strategy -- they capture live traffic off the wire, then build statistical models simulating its features. See also http://epickrram.blogspot.co.uk/2014/07/performance-testing-at-lmax-part-two.html and http://epickrram.blogspot.co.uk/2014/08/performance-testing-at-lmax-part-three.html .
performance testing tests simulation latency lmax trading sniffing packet-capture
The Violence of Algorithms: Why Big Data Is Only as Smart as Those Who Generate It
The modern state system is built on a bargain between governments and citizens. States provide collective social goods, and in turn, via a system of norms, institutions, regulations, and ethics to hold this power accountable, citizens give states legitimacy. This bargain created order and stability out of what was an increasingly chaotic global system. If algorithms represent a new ungoverned space, a hidden and potentially ever-evolving unknowable public good, then they are an affront to our democratic system, one that requires transparency and accountability in order to function. A node of power that exists outside of these bounds is a threat to the notion of collective governance itself. This, at its core, is a profoundly undemocratic notion—one that states will have to engage with seriously if they are going to remain relevant and legitimate to their digital citizenry who give them their power.
palantir algorithms big-data government democracy transparency accountability analytics surveillance war privacy protest rights
Dong detection in LEGO Universe
great example of how Minecraft solved the problem the easy way -- by simply not making an MMO, the whole problem effectively goes away
penis funny games lego lego-universe minecraft gaming mmo ugc
HTTP/2 is here, let's optimize! - Velocity SC 2015 - Google Slides
Changes which server-side developers will need to start considering as HTTP/2 rolls out. Remove domain sharding; stop concatenating resources; stop inlining resources; use server push.
Five different ways to handle leap seconds with NTP
Without switching to chronyd, ntpd -x sounds not too suboptimal:
With ntpd, the kernel backward step is used by default. With ntpd versions before 4.2.6, or 4.2.6 and later patched for this bug, the -x option (added to /etc/sysconfig/ntpd) can be used to disable the kernel leap second correction and ignore the leap second as far as the local clock is concerned. The one-second error gained after the leap second will be measured and corrected later by slewing in normal operation using NTP servers which already corrected their local clocks.
It's all pretty messy though :(ntpd ntp chronyd clocks time synchronization via:fanf linux leap-seconds
-
Russia's troll farms. Ladies and gentlemen -- the future
future abuse trolls russia trolling politics social-media twitter facebook
Ireland's media silenced over MP's speech about Denis O'Brien
this is appalling. And of course we can only find out about it from overseas media because our own media is quaking in their boots :(
media ireland he-who-cannot-be-named censorship omgwtfbbq law libel injunctions high-court
How Ireland's same-sex marriage referendum played out on Twitter
nice clear data there
-
I was in the middle of writing a breakdown of what went wrong, but you've beat me to it. Basically, they have a LinuxSecureRandom class that's supposed to override the standard SecureRandom. This class reads from /dev/urandom and should provide cryptographically secure random values. They also seed the generator using SecureRandom#setSeed with data pulled from random.org. With their custom SecureRandom, this is safe because it mixes the entropy using XOR, so even if the random.org data is dodgy it won't reduce security. It's just an added bonus. BUT! On some devices under some circumstances, the LinuxSecureRandom class doesn't get registered. This is likely because /dev/urandom doesn't exist or can't be accessed for some reason. Instead of screaming bloody murder like any sensible implementation would, they just ignore that and fall back to using the standard SecureRandom. If the above happens, there's a problem because the default implementation of SecureRandom#setSeed doesn't mix. If you set the seed, it replaces the entropy entirely. So now the entropy is coming solely from random.org. And the final mistake: They were using HTTP instead of HTTPS to make the webservice call to random.org. On Jan 4, random.org started enforcing HTTPS and returning a 301 Permanently Moved error for HTTP - see https://www.random.org/news/. So since that date, the entropy has actually been the error message (turned into bytes) instead of the expected 256-bit number. Using that seed, SecureRandom will generate the private key for address 1Bn9ReEocMG1WEW1qYjuDrdFzEFFDCq43F 100% of the time. Ouch. This is around the time that address first appears, so the timeline matches. I haven't had a thorough look at what they've replaced it with in the latest version, but initial impressions are that it's not ideal. Not disastrous, but not good.
Always check return values; always check HTTP status codes.bugs android fail securerandom random prng blockchain.info bitcoin http randomness entropy error-checking
-
A strongly specified, highly compatible implementation of Markdown
reference markdown commonmark specs formatting text compatibility
-
'A Decentralized GitHub'. nifty
I Fooled Millions Into Thinking Chocolate Helps Weight Loss
"Slim by Chocolate!" the headlines blared. A team of German researchers had found that people on a low-carb diet lost weight 10 percent faster if they ate a chocolate bar every day. It made the front page of Bild, Europe's largest daily newspaper, just beneath their update about the Germanwings crash. From there, it ricocheted around the internet and beyond, making news in more than 20 countries and half a dozen languages. It was discussed on television news shows. It appeared in glossy print, most recently in the June issue of Shape magazine ("Why You Must Eat Chocolate Daily", page 128). Not only does chocolate accelerate weight loss, the study found, but it leads to healthier cholesterol levels and overall increased well-being. The Bild story quotes the study's lead author, Johannes Bohannon, Ph.D., research director of the Institute of Diet and Health: "The best part is you can buy chocolate everywhere." I am Johannes Bohannon, Ph.D. Well, actually my name is John, and I'm a journalist. I do have a Ph.D., but it's in the molecular biology of bacteria, not humans. The Institute of Diet and Health? That's nothing more than a website. Other than those fibs, the study was 100 percent authentic. My colleagues and I recruited actual human subjects in Germany. We ran an actual clinical trial, with subjects randomly assigned to different diet regimes. And the statistically significant benefits of chocolate that we reported are based on the actual data. It was, in fact, a fairly typical study for the field of diet research. Which is to say: It was terrible science. The results are meaningless, and the health claims that the media blasted out to millions of people around the world are utterly unfounded.
Interesting bit: the online commenters commenting on the published stories quickly saw through the bullshit. Why can't the churnalising journos do that?chocolate journalism science diet food churnalism pr bild health clinical-trials papers peer-review research
-
mainly interesting for the dataviz and the Google-Doc-driven backend. wish they published the script though
google snake-oil superfoods food dataviz bubble-race-chart graphics infographics google-docs spreadsheets
Three Questions to Answer When Reporting an Error
Very long, but tl;dr:
the trick to creating an effective error message is to answer the 3 Questions within your message: What is the error? What was the probable cause of the error? What is the probable remedy?
-
Grim meathook future, courtesy of Volvo:
"The Volvo XC60 comes with City Safety as a standard feature however this does not include the Pedestrian detection functionality [...] The pedestrian detection feature [...] costs approximately $3,000.
However, there's another lesson here, in crappy car UX and the risks thereof:But even if it did have the feature, Larsson says the driver would have interfered with it by the way they were driving and "accelerating heavily towards the people in the video." "The pedestrian detection would likely have been inactivated due to the driver inactivating it by intentionally and actively accelerating," said Larsson. "Hence, the auto braking function is overrided by the driver and deactivated." Meanwhile, the people in the video seem to ignore their instincts and trust that the car assumed to be endowed with artificial intelligence knows not to hurt them. It is a sign of our incredible faith in the power of technology, but also, it's a reminder that companies making AI-assisted vehicles need to make safety features standard and communicate clearly when they aren't.
self-driving-cars cars ai pedestrian computer-vision volvo fail accidents grim-meathook-future
iPhone UTF-8 text vulnerability
'Due to how the banner notifications process the Unicode text. The banner briefly attempts to present the incoming text and then "gives up" thus the crash'. Apparently the entire Springboard launcher crashes.
apple vulnerability iphone utf-8 unicode fail bugs springboard ios via:abetson
Schedule Recurring AWS Lambda Invocations With The Unreliable Town Clock (UTC)
The Unreliable Town Clock (UTC) is a new, free, public SNS Topic (Amazon Simple Notification Service) that broadcasts a "chime" message every quarter hour to all subscribers. It can send the chimes to AWS Lambda functions, SQS queues, and email addresses. You can use the chime attributes to run your code every fifteen minutes, or only run your code once an hour (e.g., when minute == "00") or once a day (e.g., when hour == "00" and minute == "00") or any other series of intervals. You can even subscribe a function you only want to run only once at a specific time in the future: Have the function ignore all invocations until it's after the time it wants. When it is time, it can perform its job, then unsubscribe itself from the SNS Topic.
alestic aws lambda cron time clock periodic-tasks recurrence hacks
-
Soylent's not purchased by the Mark Zuckerbergs or the Larry Pages or the other tech aristocrats [...] Rather, it's been taken up by white-collar workers and students destined for perpetual toil in the digital mills. Their embrace of life hacking represents the internalisation of management practices by the managed themselves.
life-hacks soylent food politics taylorism efficiency capitalism work life
-
some good Spark optimization tips
spark performance optimization rdd emr big-data cloudera tips akka
Elements of Scale: Composing and Scaling Data Platforms
Great, encyclopedic blog post rounding up common architectural and algorithmic patterns using in scalable data platforms. Cut out and keep!
architecture storage databases data big-data scaling scalability ben-stopford cqrs druid parquet columnar-stores lambda-architecture
ISIS vs. 3D Printing | Motherboard
Morehshin Allahyari, an Iranian born artist, educator, and activist [....] is working on digitally fabricating [the] sculptures [ISIS destroyed] for a series called "Material Speculation" as part of a residency in Autodesk's Pier 9 program. The first in the series is "Material Speculation: ISIS," which, through intense research, is modeling and reproducing statues destroyed by ISIS in 2015. Allahyari isn't just interested in replicating lost objects but making it possible for anyone to do the same: Embedded within each semi-translucent copy is a flash drive with Allahyari's research about the artifacts, and an online version is coming. In this way, "Material Speculation: ISIS," is not purely a metaphorical affront to ISIS, but a practical one as well. Allahyari's work is similar to conservation efforts, including web-based Project Mosul, a small team and group of volunteers that are three-dimensionally modeling ISIS-destroyed artifacts based on crowd-sourced photographs. "Thinking about 3D printers as poetic and practical tools for digital and physical archiving and documenting has been a concept that I've been interested in for the last three years," Allahyari says. Once she began exploring the works, she discovered a thorough lack of documentation. Her research snowballed. "It became extremely important for me to think about ways to gather this information and save them for both current and future civilizations."
3d-printing fabrication scanning isis niniveh iraq morehshin-allahyari history preservation archives archival
-
great intro
kubernetes ops docker containers rocket deployment packaging
A Piece of Apple II History Cracks Open — May 24, 2015
Lovely description of cracking (ie. copy-protection removal) in the Apple-II era. Very reminiscent of the equivalent in the C=64 scene, from my experience. ;)
history c=64 apple-ii personal-computers archive cracks copy-protection hacking
Deploying Elastic Beanstalk Applications from Docker Containers - Elastic Beanstalk
oh wow, this actually sounds pretty cool
TIL we have more gravity than Canada
'Early gravity mapping efforts in the 1960s revealed that the Hudson Bay area in particular exerts a weaker gravitational force. Since less mass equals less gravity, there must be less mass underneath these areas.' informed!
SolarCapture Packet Capture Software
Interesting product line -- I didn't know this existed, but it makes good sense as a "network flight recorder". Big in finance.
SolarCapture is powerful packet capture product family that can transform every server into a precision network monitoring device, increasing network visibility, network instrumentation, and performance analysis. SolarCapture products optimize network monitoring and security, while eliminating the need for specialized appliances, expensive adapters relying on exotic protocols, proprietary hardware, and dedicated networking equipment.
See also Corvil (based in Dublin!): 'I'm using a Corvil at the moment and it's awesome- nanosecond precision latency measurements on the wire.' (via mechanical sympathy list)corvil timing metrics measurement latency network solarcapture packet-capture financial performance security network-monitoring
Top 10 data mining algorithms in plain English
This is a phenomenally useful ML/data-mining resource post -- 'the top 10 most influential data mining algorithms as voted on by 3 separate panels in [ICDM '06's] survey paper', but with a nice clear intro and description for each one. Here's the algorithms covered:
1. C4.5 2. k-means 3. Support vector machines 4. Apriori 5. EM 6. PageRank 7. AdaBoost 8. kNN 9. Naive Bayes 10. CART
svm k-means c4.5 apriori em pagerank adaboost knn naive-bayes cart ml data-mining machine-learning papers algorithms unsupervised supervised
Developer believes he can turn digital game into global hit
g'wan the Colm!
Trend Micro Locality Sensitive Hash
a fuzzy matching library. Given a byte stream with a minimum length of 512 bytes, TLSH generates a hash value which can be used for similarity comparisons. Similar objects will have similar hash values which allows for the detection of similar objects by comparing their hash values. Note that the byte stream should have a sufficient amount of complexity. For example, a byte stream of identical bytes will not generate a hash value.
Paper here: https://drive.google.com/file/d/0B6FS3SVQ1i0GTXk5eDl3Y29QWlk/edit via adulaunilsimsa sdhash ssdeep locality-sensitive hashing algorithm hashes trend-micro tlsh hash fuzzy-matching via:adulau
Eric Brewer interview on Kubernetes
What is the relationship between Kubernetes, Borg and Omega (the two internal resource-orchestration systems Google has built)? I would say, kind of by definition, there's no shared code but there are shared people. You can think of Kubernetes?—?especially some of the elements around pods and labels?—?as being lessons learned from Borg and Omega that are, frankly, significantly better in Kubernetes. There are things that are going to end up being the same as Borg?—?like the way we use IP addresses is very similar?—?but other things, like labels, are actually much better than what we did internally. I would say that's a lesson we learned the hard way.
google architecture kubernetes docker containers borg omega deployment ops
'Can People Distinguish Pâté from Dog Food?'
Ugh.
Considering the similarity of its ingredients, canned dog food could be a suitable and inexpensive substitute for pâté or processed blended meat products such as Spam or liverwurst. However, the social stigma associated with the human consumption of pet food makes an unbiased comparison challenging. To prevent bias, Newman's Own dog food was prepared with a food processor to have the texture and appearance of a liver mousse. In a double-blind test, subjects were presented with five unlabeled blended meat products, one of which was the prepared dog food. After ranking the samples on the basis of taste, subjects were challenged to identify which of the five was dog food. Although 72% of subjects ranked the dog food as the worst of the five samples in terms of taste (Newell and MacFarlane multiple comparison, P<0.05), subjects were not better than random at correctly identifying the dog food.
pate food omgwtf science research dog-food meat economics taste flavour
Redditor runs the secret Python code in Ex Machina
and finds:
when you run with python2.7 you get the following: ISBN = 9780199226559 Which is Embodiment and the inner life: Cognition and Consciousness in the Space of Possible Minds. and so now I have a lot more respect for the Director.
-
via the Dublin Ladies Beer Society ;)
-
major bug in kernel versions 3.14 - 3.18 on Haswell hardware
-
'The multiple repository tool'. How Google kludged around the split-repo problem when you don't have a monorepo.
kludges git monorepo monorepi google android aosp repo coding version-control dvcs
Declaratively Provision Docker Images Using Nix
I really wish Docker/CoreOS would look at copying some of the deterministic-build ideas from Nix; see also http://gregoryszorc.com/blog/2014/10/13/deterministic-and-minimal-docker-images/
build packaging docker nix nix-docker deterministic-builds nixos apollo brazil
Please stop calling databases CP or AP
In his excellent blog post [...] Jeff Hodges recommends that you use the CAP theorem to critique systems. A lot of people have taken that advice to heart, describing their systems as "CP" (consistent but not available under network partitions), "AP" (available but not consistent under network partitions), or sometimes "CA" (meaning "I still haven't read Coda's post from almost 5 years ago"). I agree with all of Jeff's other points, but with regard to the CAP theorem, I must disagree. The CAP theorem is too simplistic and too widely misunderstood to be of much use for characterizing systems. Therefore I ask that we retire all references to the CAP theorem, stop talking about the CAP theorem, and put the poor thing to rest. Instead, we should use more precise terminology to reason about our trade-offs.
cap databases storage distcomp ca ap cp zookeeper consistency reliability networking
-
Non-monospaced coding fonts! I'm all in favour...
As writing and managing code becomes more complex, today's sophisticated coding environments are evolving to include everything from breakpoint markers to code folding and syntax highlighting. The typography of code should evolve as well, to explore possibilities beyond one font style, one size, and one character width.
input fonts via:its typography code coding font text ide monospace
-
a Zipkin-compatible distributed-system tracing framework in Java, in the Apache Incubator
Intel speeds up etcd throughput using ADR Xeon-only hardware feature
To reduce the latency impact of storing to disk, Weaver's team looked to buffering as a means to absorb the writes and sync them to disk periodically, rather than for each entry. Tradeoffs? They knew memory buffers would help, but there would be potential difficulties with smaller clusters if they violated the stable storage requirement. Instead, they turned to Intel's silicon architects about features available in the Xeon line. After describing the core problem, they found out this had been solved in other areas with ADR. After some work to prove out a Linux OS supported use for this, they were confident they had a best-of-both-worlds angle. And it worked. As Weaver detailed in his CoreOS Fest discussion, the response time proved stable. ADR can grab a section of memory, persist it to disk and power it back. It can return entries back to disk and restore back to the buffer. ADR provides the ability to make small (<100MB) segments of memory "stable" enough for Raft log entries. It means it does not need battery-backed memory. It can be orchestrated using Linux or Windows OS libraries. ADR allows the capability to define target memory and determine where to recover. It can also be exposed directly into libs for runtimes like Golang. And it uses silicon features that are accessible on current Intel servers.
kubernetes coreos adr performance intel raft etcd hardware linux persistence disk storage xeon
streamtools: a graphical tool for working with streams of data | nytlabs
Visual programming, Yahoo! Pipes style, back again:
we have created streamtools – a new, open source project by The New York Times R&D Lab which provides a general purpose, graphical tool for dealing with streams of data. It provides a vocabulary of operations that can be connected together to create live data processing systems without the need for programming or complicated infrastructure. These systems are assembled using a visual interface that affords both immediate understanding and live manipulation of the system.
via Aman-
a Java based low latency, high throughput message bus, built on top of a memory mapped file; inspired by Java Chronicle with the main difference that it's designed to efficiently support multiple writers – enabling use cases where the order of messages produced by multiple processes are important. MappedBus can be also described as an efficient IPC mechanism which enable several Java programs to communicate by exchanging messages.
ipc java jvm mappedbus low-latency mmap message-bus data-structures queue message-passing
Amazon's Drone Delivery Patent Just Feels Like Trolling At This Point
Oh dear, Amazon.
These aren't actual technologies yet. [...] All of which underscores that Amazon might never ever ever ever actually implement delivery drones. The patent paperwork was filed nearly a year after Amazon's splashy drone program reveal on 60 Minutes. At the time we called it revolutionary marketing because, you know, delivery drones are technical and logistical madness, not to mention that commercial drone use is illegal right now. Although, in fairness the FAA did just relax some rules so that Amazon could test drones. At this point it feels like Amazon is just trolling. It's trolling us with public relations BS about its future drones, and it's trolling future competitors -- Google is also apparently working on this -- so that if somebody ever somehow does anything relating to drone delivery, Amazon can sue them. If I'm wrong, I'll deliver my apology via Airmail.
amazon trolling patents uspto delivery drones uavs competition faa
-
This is like watching a train-wreck in slow motion on Groundhog Day. We, in the broader Linux and open source community, have been down this path multiple times over the past fifteen years, specifically with package formats. While there needs to be room for experimentation, having two incompatible specs driven by two startups trying to differentiate and in direct competition is *not* a good thing. It would be better for the community and for everyone who depends on our collective efforts if CoreOS and Docker collaborated on a standardized common spec, image format, and distribution protocol. To this end, we at Red Hat will continue to contribute to both initiatives with the goal of driving convergence.
rkt docker appc coreos red-hat dpkg rpm linux packaging collaboration open-source
Migration to, Expectations, and Advanced Tuning of G1GC
Bookmarking for future reference. recommended by one of the GC experts, I can't recall exactly who ;)
Deploy a registry - Docker Documentation
Looks like it's pretty feasible to run a private Docker registry on every host, backed by S3 (according to the ECS team's AMA). SPOF-free -- handy
How to change Gradle cache location
$GRADLE_USER_HOME, basically -- it may also be possible to set from the Gradle script itself too
-
"An archive of the former Internet of Things"
archive iot things internet nabaztag startups acquisitions tumblr gadgets history
Memory Layouts for Binary Search
Key takeaway:
Nearly universally, B-trees win when the data gets big enough.
caches cpu performance optimization memory binary-search b-trees algorithms search memory-layout
Understanding the Docker Cache for Faster Builds
good advice. see also the Best Practices official doc at https://docs.docker.com/articles/dockerfile_best-practices/
Your Google Algorithm Cheat Sheet: Panda, Penguin, and Hummingbird
Interesting that GOOG are still doing these big-bang releases -- I guess crunching the data to come up with new weights/rules is a heavyweight, time-consuming process
google search ranking releases panda penguin hummingbird weighting
Dublin Bike Theft Survey Results
Dublin Cycling Campaign's survey results: estimated 20,000 bikes stolen per year in Dublin; only 1% of thefts results in a conviction
dublin bikes cycling theft crime statistics infographics dcc
DRUG PUMP'S SECURITY FLAW LETS HACKERS RAISE DOSE LIMITS
The Hospira drug pump vulnerabilities described here sound pretty horrific
drugs drug-pumps hospira exploits vulnerabilities security root dosage limits
-
+1 to ALL of this. We are doing exactly the same in Swrve and it has radically improved our release quality
How to do named entity recognition: machine learning oversimplified
Good explanation of this NLP tokenization/feature-extraction technique. Example result: "Jimi/B-PER Hendrix/I-PER played/O at/O Woodstock/B-LOC ./O"
named-entities feature-extraction tokenization nlp ml algorithms machine-learning
The Discovery of Apache ZooKeeper's Poison Packet - PagerDuty
Excellent deep dive into a production issue. Root causes: crappy error handling code in Zookeeper; lack of bounds checking in ZK; and a nasty kernel bug.
zookeeper bugs error-handling bounds-checking oom poison-packets pagerduty packets tcpdump xen aes linux kernel
The Injector: A new Executor for Java
This honestly fits a narrow niche, but one that is gaining in popularity. If your messages take > 100?s to process, or your worker threads are consistently saturated, the standard ThreadPoolExecutor is likely perfectly adequate for your needs. If, on the other hand, you're able to engineer your system to operate with one application thread per physical core you are probably better off looking at an approach like the LMAX Disruptor. However, if you fall in the crack in between these two scenarios, or are seeing a significant portion of time spent in futex calls and need a drop in ExecutorService to take the edge off, the injector may well be worth a look.
performance java executor concurrency disruptor algorithms coding threads threadpool injector
-
Excellent mobile-phone plan comparison site for the Irish market, using apps which you install and which analyse your call history, data usage, etc. over the past month to compute the optimal plan based on your usage. Pretty amazing results in my case! The only downside is the privacy policy, which allows the company to resell your usage data (anonymised, and in aggregate) -- I'd really prefer if this wasn't the case :(
mobile-phones shopping tesco emobile 3g 4g ireland plans comparison-shopping killbiller via:its
Family in No poster Says YES to Marriage Equality | Amnesty International
Beyond the politics, the risks of stock photo usage are pretty evident too:
"In 2014, as a young family, we did a photo shoot with a photographer friend to get some nice shots for the family album. No money was exchanged – we got nice photos for free, they got nice images for their portfolio. As part of this agreement, we agreed to let them upload them to a stock photo album. We knew that these were available for purchase and we gave permission. Perhaps, naïvely, we imagined that on the off chance that any was ever selected, it might be for a small magazine or website. To confirm, we have not received any money for the photo – then or now, and nor do we expect any. We were surprised and upset to see that the photo was being used as part of a campaign with which we do not agree. We completely support same-sex marriage, and we believe that same-sex couples' should of course be able to adopt, as we believe that they are equally able to provide children with much-needed love and care. To suggest otherwise is offensive to us, and to many others."
ssm ireland politics amnesty stock-photos ip rights photos campaigns ads
Lambda: Bees with Frickin' Laser Beams
a HTTP testing tool in AWS Lambda. nice enough, but still a toy...
-
Some good factoids about Loggly's Kafka usage and scales
Patterns for building a resilient and scalable microservices platform on AWS
Some good details from Boyan Dimitrov at Hailo, on their orchestration, deployment, provisioning infra they've built
deployment ops devops hailo microservices platform patterns slides
-
A probabilistic data structure for frequency/k-occurrence cardinality estimation of multisets. Sample implementation
(via Patrick McFadin)via:patrickmcfadin hyperloglog cardinality data-structures algorithms hyperlogsandwich counting estimation lossy multisets
"Trash Day: Coordinating Garbage Collection in Distributed Systems"
Another GC-coordination strategy, similar to Blade (qv), with some real-world examples using Cassandra
blade via:adriancolyer papers gc distsys algorithms distributed java jvm latency spark cassandra
Five Takeaways on the State of Natural Language Processing
Good overview of the state of the art in NLP nowadays. I particularly like word2vec interesting:
Embedding words as real-numbered vectors using a skip-gram, negative-sampling model (word2vec code) was mentioned in nearly every talk I attended. Either companies are using various word2vec implementations directly or they are building diffs off of the basic framework. Trained on large corpora, the vector representations encode concepts in a large dimensional space (usually 200-300 dim).
Quite similar to some tokenization approaches we experimented with in SpamAssassin, so I don't find this too surprising....word2vec nlp tokenization machine-learning language parsing doc2vec skip-grams data-structures feature-extraction via:lemonodor
Smarter testing Java code with Spock Framework
hmm, looks quite nice as a potential next-gen JUnit replacement for unit tests
-
'Baby Friendly Holidays | Child, Toddler & Family Villas | France | Spain | Portugal | Italy'. Joe swears by it, will give it a go next year
How the NSA Converts Spoken Words Into Searchable Text - The Intercept
This hits the nail on the head, IMO:
To Phillip Rogaway, a professor of computer science at the University of California, Davis, keyword-search is probably the "least of our problems." In an email to The Intercept, Rogaway warned that "When the NSA identifies someone as 'interesting' based on contemporary NLP methods, it might be that there is no human-understandable explanation as to why beyond: 'his corpus of discourse resembles those of others whom we thought interesting'; or the conceptual opposite: 'his discourse looks or sounds different from most people's.' If the algorithms NSA computers use to identify threats are too complex for humans to understand, it will be impossible to understand the contours of the surveillance apparatus by which one is judged. All that people will be able to do is to try your best to behave just like everyone else."
privacy security gchq nsa surveillance machine-learning liberty future speech nlp pattern-analysis cs
awslabs/aws-lambda-redshift-loader
Load data into Redshift from S3 buckets using a pre-canned Lambda function. Looks like it may be a good example of production-quality Lambda
-
'Aerospike offers phenomenal latencies and throughput -- but in terms of data safety, its strongest guarantees are similar to Cassandra or Riak in Last-Write-Wins mode. It may be a safe store for immutable data, but updates to a record can be silently discarded in the event of network disruption. Because Aerospike's timeouts are so aggressive–on the order of milliseconds -- even small network hiccups are sufficient to trigger data loss. If you are an Aerospike user, you should not expect "immediate", "read-committed", or "ACID consistency"; their marketing material quietly assumes you have a magical network, and I assure you this is not the case. It's certainly not true in cloud environments, and even well-managed physical datacenters can experience horrible network failures.'
aerospike outages cap testing jepsen aphyr databases storage reliability
Emojineering Part 1: Machine Learning for Emoji Trends - Instagram Engineering
Instagram figuring out meanings from Emoji usage contexts using ML. ????
instagram emoji cool language text internet web speech communication trends machine-learning analysis
Call me maybe: Elasticsearch 1.5.0
tl;dr: Elasticsearch still hoses data integrity on partition, badly
elasticsearch reliability data storage safety jepsen testing aphyr partition network-partitions cap
In the privacy of your own home
I didn't know about this:
Last spring, as 41,000 runners made their way through the streets of Dublin in the city's Women's Mini Marathon, an unassuming redheaded man by the name of Candid Wueest stood on the sidelines with a scanner. He had built it in a couple of hours with $75 worth of parts, and he was using it to surreptitiously pick up data from activity trackers worn on the runners' wrists. During the race, Wueest managed to collect personal info from 563 racers, including their names, addresses, and passwords, as well as the unique IDs of the devices they were carrying.
dublin candid-wueest privacy data marathon running iot activity-trackers
David P. Reed on the history of UDP
'UDP was actually "designed" in 30 minutes on a blackboard when we decided pull the original TCP protocol apart into TCP and IP, and created UDP on top of IP as an alternative for multiplexing and demultiplexing IP datagrams inside a host among the various host processes or tasks. But it was a placeholder that enabled all the non-virtual-circuit protocols since then to be invented, including encapsulation, RTP, DNS, …, without having to negotiate for permission either to define a new protocol or to extend TCP by adding "features".'
Oops: Instagram forgot to renew its SSL certificate
hooray for cert renewal pain
certs ssl renewal expiry instagram outages lifecycle web https
-
Seth Vargo is correct. Its not the bit length of the key which is at issue, its the signature algorithm. The entire keychain for the s3.awsamazon.com key is signed with SHA1withRSA: https://www.ssllabs.com/ssltest/analyze.html?d=s3.amazonaws.com&s=54.231.244.0&hideResults=on At issue is that the root verisign key has been marked as weak because of SHA1 and taken out of the curl bundle which is widely popular, and this issue will continue to cause more and more issues going forwards as that bundle makes it way into shipping o/s distributions and aws certification verification breaks.
'This is still happening and curl is now failing on my machine causing all sorts of fun issues (including breaking CocoaPods that are using S3 for storage).' -- @jmhodges This may be a contributory factor to the issue @nelson saw: https://nelsonslog.wordpress.com/2015/04/28/cyberduck-is-responsible-for-my-bad-ssl-certificate/ Curl's ca-certs bundle is also used by Node: https://github.com/joyent/node/issues/8894 and doubtless many other apps and packages. Here's a mailing list thread discussing the issue: http://curl.haxx.se/mail/archive-2014-10/0066.html -- looks like the curl team aren't too bothered about it.curl s3 amazon aws ssl tls certs sha1 rsa key-length security cacerts
Cassandra moving to using G1 as the default recommended GC implementation
This is a big indicator that G1 is ready for primetime. CMS has long been the go-to GC for production usage, but requires careful, complex hand-tuning -- if G1 is getting to a stage where it's just a case of giving it enough RAM, that'd be great. Also, looks like it'll be the JDK9 default: https://twitter.com/shipilev/status/593175793255219200
cassandra tuning ops g1gc cms gc java jvm production performance memory
-
ThisIsColossal now have a shop! bookmarking for some lovely gifts
Eight lessons learned hacking on GitHub Pages for six months
Pages is actually pretty solid -- nice one GitHub
-
Static code analysis for shell scripts (via Tony Finch)
-
presentation from last week's Craft Conference in Budapest; Tammer Saleh of Pivotal with a few antipatterns observed in dealing with microservices.
microservices soa architecture design coding software presentations slides tammer-saleh pivotal craft
-
'a command line tool that (hopefully) makes it easier to deploy, update, and test functions for AWS Lambda.' much needed IMO -- Lambda is too closed
-
HashiCorp's take on the secrets-storage system. looks good
hashicorp deployment security secrets authentication vault storage keys key-rotation
Everything Science Knows Right Now About Standing Desks | Co.Design
"Overall, current evidence suggests that both standing and treadmill desks may be effective in improving overall health considering both physiological and mental health components."
standing-desks treadmill-desks desks exercise health work workplace back sitting standing
Race conditions on Facebook, DigitalOcean and others
good trick -- exploit eventual consistency and a lack of distributed transactions by launching race-condition-based attacks
attacks exploits race-conditions bugs eventual-consistency distributed-transactions http facebook digitalocean via:aphyr
-
'Discover and discuss the best dev tools and cloud infrastructure services' -- fun!
stackshare architecture stack ops software ranking open-source
-
a web-based SSH console that centrally manages administrative access to systems. Web-based administration is combined with management and distribution of user's public SSH keys. Key management and administration is based on profiles assigned to defined users. Administrators can login using two-factor authentication with FreeOTP or Google Authenticator . From there they can create and manage public SSH keys or connect to their assigned systems through a web-shell. Commands can be shared across shells to make patching easier and eliminate redundant command execution.
32-bit overflow in BitGo js code caused an accidental 85 BTC transaction fee
Yes, this is a fucking 32-bit integer overflow. Whatever software was used, it calculated the sum of all inputs using 32-bit variables, which overflow at about 20 BTC if signed or 40 BTC if not. The fee was supposed to be 0xC350 = 50,000 satoshis, but it turned out to be 0x2,0000,C350 = 8,589,984,592 satoshis. Captains of the industry. If they were captains of any other industry, like say for example automotive, we'd have people dying in car crashes between two stationary vehicles.
Eight Docker Development Patterns
good Docker tips
-
We hope this report helps to round out the overall facts known about this attack. It also demonstrates that collectively there is a lot of visibility into what happens on the web. At the HTTP level seen by Safe Browsing, we cannot confidently attribute this attack to anyone. However, it makes it clear that hiding such attacks from detailed analysis after the fact is difficult. Had the entire web already moved to encrypted traffic via TLS, such an injection attack would not have been possible. This provides further motivation for transitioning the web to encrypted and integrity-protected communication. Unfortunately, defending against such an attack is not easy for website operators. In this case, the attack Javascript requests web resources sequentially and slowing down responses might have helped with reducing the overall attack traffic. Another hope is that the external visibility of this attack will serve as a deterrent in the future.
Via Nelson.google security via:nelson ddos javascript tls ssl safe-browsing networking china greatfire
Amazon EC2 Container Service team AmA
a few answers here. Mostly people pointing out shortcomings and the team asking them to start a thread on their forum though :(
Cluster-Based Architectures Using Docker and Amazon EC2 Container Service
In this post, we're going to take a deeper dive into the architectural concepts underlying cluster computing using container management frameworks such as ECS. We will show how these frameworks effectively abstract the low-level resources such as CPU, memory, and storage, allowing for highly efficient usage of the nodes in a compute cluster. Building on some of the concepts detailed in the earlier posts, we will discover why containers are such a good fit for this type of abstraction, and how the Amazon EC2 Container Service fits into the larger ecosystem of cluster management frameworks.
-
'Here are four Kubernetes features that came from our experiences with Borg.'
attacks using U+202E - RIGHT-TO-LEFT OVERRIDE
Security implications of in-band signalling strikes again, 43 years after the "Blue Box" hit the mainstream. Jamie McCarthy on Twitter: ".@cmdrtaco - Remember when we had to block the U+202E code point in Slashdot comments to stop siht ekil stnemmoc? https://t.co/TcHxKkx9Oo" See also http://krebsonsecurity.com/2011/09/right-to-left-override-aids-email-attacks/ -- GMail was vulnerable too; and http://en.wikipedia.org/wiki/Unicode_control_characters for more inline control chars. http://unicode.org/reports/tr36/#Bidirectional_Text_Spoofing has some official recommendations from the Unicode consortium on dealing with bidi override chars.
security attacks rlo unicode control-characters codepoints bidi text gmail slashdot sanitization input
Meet the e-voting machine so easy to hack, it will take your breath away | Ars Technica
The AVS WinVote system -- mind-bogglingly shitty security.
If an election was held using the AVS WinVote, and it wasn't hacked, it was only because no one tried. The vulnerabilities were so severe, and so trivial to exploit, that anyone with even a modicum of training could have succeeded. They didn't need to be in the polling place—within a few hundred feet (e.g., in the parking lot) is easy, and within a half mile with a rudimentary antenna built using a Pringles can. Further, there are no logs or other records that would indicate if such a thing ever happened, so if an election was hacked any time in the past, we will never know. I've been in the security field for 30 years, and it takes a lot to surprise me. But the VITA report really shocked me—as bad as I thought the problems were likely to be, VITA's five-page report showed that they were far worse. And the WinVote system was so fragile that it hardly took any effort. While the report does not state how much effort went into the investigation, my estimation based on the description is that it was less than a person week.
security voting via:johnke winvote avs shoup wep wifi windows
'Continuous Deployment: The Dirty Details'
Good slide deck from Etsy's Mike Brittain regarding their CD setup. Some interesting little-known details: Slide 41: database schema changes are not CD'd -- they go out on "Schema change Thursdays". Slide 44: only the webapp is CD'd -- PHP, Apache, memcache components (Etsy.com, support and back-office tools, developer API, gearman async worker queues). The external "services" are not -- databases, Solr/JVM search (rolling restarts), photo storage (filters, proxy cache, S3), payments (PCI-DSS, controlled access). They avoid schema changes and breaking changes using an approach they call "non-breaking expansions" -- expose new version in a service interface; support multiple versions in the consumer. Example from slides 50-63, based around a database schema migration. Slide 66: "dev flags" (rollout oriented) are promoted to "feature flags" (long lived degradation control). Slide 71: some architectural philosophies: deploying is cheap; releasing is cheap; gathering data should be cheap too; treat first iterations as experiments. Slide 102: "Canary pools". They have multiple pools of users for testing in production -- the staff pool, users who have opted in to see prototypes/beta stuff, 0-100% gradual phased rollout.
cd deploy etsy slides migrations database schema ops ci version-control feature-flags
Etsy's Release Management process
Good info on how Etsy use their Deployinator tool, end-to-end. Slide 11: git SHA is visible for each env, allowing easy verification of what code is deployed. Slide 14: Code is deployed to "princess" staging env while CI tests are running; no need to wait for unit/CI tests to complete. Slide 23: smoke tests of pre-prod "princess" (complete after 8 mins elapsed). Slide 31: dashboard link for deployed code is posted during deploy; post-release prod smoke tests are run by Jenkins. (short ones! they complete in 42 seconds)
deployment etsy deploy deployinator princess staging ops testing devops smoke-tests production jenkins
Makerbot's Saddest Hour | TechCrunch
I've been speaking to a few people [at Makerbot] who prefer to remain anonymous and most of my contacts there are gone (the head of PR was apparently fired) and don't want to talk. But the new from inside is troubling. The mass-layoffs are blamed on low revenue and one former employee wrote "Company was failing. Couldn't pay vendors, had to downsize." Do I think Makerbot will sink? At this point I don't know.
-
'CredStash is a very simple, easy to use credential management and distribution system that uses AWS Key Management System (KMS) for key wrapping and master-key storage, and DynamoDB for credential storage and sharing.'
aws credstash python security keys key-management secrets kms
ferd.ca -> Lessons Learned while Working on Large-Scale Server Software
Good advice
distributed scalability systems coding server-side erlang devops networking reliability
Internet Scale Services Checklist
good aspirational checklist, inspired heavily by James Hamilton's seminal 2007 paper, "On Designing And Deploying Internet-Scale Services"
james-hamilton checklists ops internet-scale architecture operability monitoring reliability availability uptime aspirations
FBI admits flaws in hair analysis over decades
Wow, this is staggering.
The Justice Department and FBI have formally acknowledged that nearly every examiner in an elite FBI forensic unit gave flawed testimony in almost all trials in which they offered evidence against criminal defendants over more than a two-decade period before 2000. [....] The review confirmed that FBI experts systematically testified to the near-certainty of "matches" of crime-scene hairs to defendants, backing their claims by citing incomplete or misleading statistics drawn from their case work. In reality, there is no accepted research on how often hair from different people may appear the same. Since 2000, the lab has used visual hair comparison to rule out someone as a possible source of hair or in combination with more accurate DNA testing. Warnings about the problem have been mounting. In 2002, the FBI reported that its own DNA testing found that examiners reported false hair matches more than 11 percent of the time.
fbi false-positives hair dna biometrics trials justice experts crime forensics inaccuracy csi
-
Most or all of the missing bitcoins were stolen straight out of the MtGox hot wallet over time, beginning in late 2011. As a result, MtGox operated at fractional reserve for years (knowingly or not), and was practically depleted of bitcoins by 2013. A significant number of stolen bitcoins were deposited onto various exchanges, including MtGox itself, and probably sold for cash (which at the bitcoin prices of the day would have been substantially less than the hundreds of millions of dollars they were worth at the time of MtGox's collapse). MtGox' bitcoins continuously went missing over time, but at a decreasing pace. Again by the middle of 2013, the curve goes more or less flat, matching the hypothesis that by that time there may not have been any more bitcoins left to lose. The rate of loss otherwise seems unusually smooth and at the same time not strictly relative to any readily available factors such as remaining BTC holdings, transaction volumes or the BTC price. Worth pointing out is that, thanks to having matched up most of the deposit/withdrawal log earlier, we can at this point at least rule out the possibility of any large-scale fake deposits — the bitcoins going into MtGox were real, meaning the discrepancy was likely rather caused by bitcoins leaving MtGox without going through valid withdrawals.
Bank of the Underworld - The Atlantic
Prosecutors analyzed approximately 500 of Liberty Reserve's biggest accounts, which constituted 44 percent of its business. The government contends that 32 of these accounts were connected to the sale of stolen credit cards and 117 were used by Ponzi-scheme operators. All of this activity flourished, prosecutors said, because Liberty Reserve made no real effort to monitor its users for criminal behavior. What's more, records showed that one of the company's top tech experts, Mark Marmilev, who was also arrested, appeared to have promoted Liberty Reserve in chat rooms devoted to Ponzi schemes.
(via Nelson)scams fraud crime currency the-atlantic liberty-reserve ponzi-schemes costa-rica arthur-budovsky banking anonymity cryptocurrency money-laundering carding
I was a Lampedusa refugee. Here's my story of fleeing Libya – and surviving
'The boy next to me fell to the floor and for a moment I didn't know if he had fainted or was dead – then I saw that he was covering his eyes so he didn't have to see the waves any more. A pregnant woman vomited and started screaming. Below deck, people were shouting that they couldn't breathe, so the men in charge of the boat went down and started beating them. By the time we saw a rescue helicopter, two days after our boat had left Libya with 250 passengers on board, some people were already dead – flung into the sea by the waves, or suffocated downstairs in the dark.'
lampedusa migration asylum europe fortress-europe italy politics immigration libya refugees
Run your own high-end cloud gaming service on EC2
Using Steam streaming and EC2 g2.2xlarge spot instances -- 'comes out to around $0.52/hr'. That's pretty compelling IMO
aws ec2 gaming games graphics spot-instances hacks windows steam
Running Arbitrary Executables in AWS Lambda
actually an officially-supported mode. huh
Exclusive: Chopra says ECB's threats to Ireland were 'outrageous' - Independent.ie
The letters urged the then-government to commit to structural reforms and restructuring of the financial sector. "That is not their job," Mr Chopra said. "Their mandate is to meet inflation. And if you lecture the ECB as to how they might go about that, they talk about their independence. "But when it comes to lecturing others about fiscal policy or structural policy, they're not at all hesitant. I'm not surprised that the people in Ireland were very upset about these letters from [Jean-Claude] Trichet."
Writing Minecraft Plugins - The Book
wow, Walter Higgins' book (from Peachpit Press) is looking great
-
Pinterest's Hadoop workflow manager; 'scalable, reliable, simple, extensible' apparently. Hopefully it allows upgrades of a workflow component without breaking an existing run in progress, like LinkedIn's Azkaban does :(
python pinterest hadoop workflows ops pinball big-data scheduling
HACKERS COULD COMMANDEER NEW PLANES THROUGH PASSENGER WI-FI
Boeing 787 Dreamliner jets, as well as Airbus A350 and A380 aircraft, have Wi-Fi passenger networks that use the same network as the avionics systems of the planes
What the fucking fuck. Air-gap or gtfoair-gap security planes boeing a380 a350 dreamliner networking firewalls avionics
Tips for debugging EC2 Container Service
some basic ECS tips from Gilt
_Blade: a Data Center Garbage Collector_
Essentially, add a central GC scheduler to improve tail latencies in a cluster, by taking instances out of the pool to perform slow GC activity instead of letting them impact live operations. I've been toying with this idea for a while, nice to see a solid paper about it
gc latency tail-latencies papers blade go java scheduling clustering load-balancing low-latency performance
SCADA systems online, and a horror story about a non-airgapped Boeing 747 engine management system
747's are big flying Unix hosts. At the time, the engine management system on this particular airline was Solaris based. The patching was well behind and they used telnet as SSH broke the menus and the budget did not extend to fixing this. The engineers could actually access the engine management system of a 747 in route. If issues are noted, they can re-tune the engine in air. The issue here is that all that separated the engine control systems and the open network was NAT based filters. There were (and as far as I know this is true today), no extrusion controls. They filter incoming traffic, but all outgoing traffic is allowed.
(via Paddy Benson)-
Nice, simple "build a website" platform. Keeping this one bookmarked for the next time someone non-techie asks me for the simplest way to do just that (thanks for the tip, Oisin)
Extracting Structured Data From Recipes Using Conditional Random Fields
nice probabilistic/ML approach to recipe parsing
nytimes recipes parsing text nlp machine-learning probabilistic crf++ algorithms feature-extraction
Large-scale cluster management at Google with Borg
Google's Borg system is a cluster manager that runs hundreds of thousands of jobs, from many thousands of different applications, across a number of clusters each with up to tens of thousands of machines. It achieves high utilization by combining admission control, efficient task-packing, over-commitment, and machine sharing with process-level performance isolation. It supports high-availability applications with runtime features that minimize fault-recovery time, and scheduling policies that reduce the probability of correlated failures. Borg simplifies life for its users by offering a declarative job specification language, name service integration, real-time job monitoring, and tools to analyze and simulate system behavior. We present a summary of the Borg system architecture and features, important design decisions, a quantitative analysis of some of its policy decisions, and a qualitative examination of lessons learned from a decade of operational experience with it.
(via Conall)via:conall clustering google papers scale to-read borg cluster-management deployment packing reliability redundancy
Keeping Your Car Safe From Electronic Thieves - NYTimes.com
In a normal scenario, when you walk up to a car with a keyless entry and try the door handle, the car wirelessly calls out for your key so you don't have to press any buttons to get inside. If the key calls back, the door unlocks. But the keyless system is capable of searching for a key only within a couple of feet. Mr. Danev said that when the teenage girl turned on her device, it amplified the distance that the car can search, which then allowed my car to talk to my key, which happened to be sitting about 50 feet away, on the kitchen counter. And just like that, open sesame.
What the hell -- who designed a system that would auto-unlock based on signal strength alone?!!security fail cars keys signal proximity keyless-entry prius toyota crime amplification power-amplifiers 3db keyless
Closed access means people die
'We've paid 100 BILLION USD over the last 10 years to "publish" science and medicine. Ebola is a massive systems failure.' See also https://www.techdirt.com/articles/20150409/17514230608/dont-think-open-access-is-important-it-might-have-prevented-much-ebola-outbreak.shtml : 'The conventional wisdom among public health authorities is that the Ebola virus, which killed at least 10,000 people in Liberia, Sierra Leone and Guinea, was a new phenomenon, not seen in West Africa before 2013. [...] But, as the team discovered, that "conventional wisdom" was wrong. In fact, they found a bunch of studies, buried behind research paywalls, that revealed that there was significant evidence of antibodies to the Ebola virus in Liberia and in other nearby nations. There was one from 1982 that noted: "medical personnel in Liberian health centers should be aware of the possibility that they may come across active cases and thus be prepared to avoid nosocomial epidemics."
deaths liberia ebola open-access papers elsevier science medicine reprints
Making Pinterest — Learn to stop using shiny new things and love MySQL
'The third reason people go for shiny is because older tech isn't advertised as aggressively as newer tech. The younger companies needs to differentiate from the old guard and be bolder, more passionate and promise to fulfill your wildest dreams. But most new tech sales pitches aren't generally forthright about their many failure modes. In our early days, we fell into this third trap. We had a lot of growing pains as we scaled the architecture. The most vocal and excited database companies kept coming to us saying they'd solve all of our scalability problems. But nobody told us of the virtues of MySQL, probably because MySQL just works, and people know about it.' It's true! -- I'm still a happy MySQL user for some use cases, particularly read-mostly relational configuration data...
Microservices and elastic resource pools with Amazon EC2 Container Service
interesting approach to working around ECS' shortcomings -- bit specific to Hailo's microservices arch and IPC mechanism though. aside: I like their version numbering scheme: ISO-8601, YYYYMMDDHHMMSS. keep it simple!
versioning microservices hailo aws ec2 ecs docker containers scheduling allocation deployment provisioning qos
Please Kill Me (Eventually) | Motherboard
There is much that the wise application of technology can do to help us ease off this mortal coil, instead of tormenting ourselves at the natural end of life in a futile, undignified and excruciating attempt to keep it somehow duct-taped on. Train more people in geriatrics, for example. Learn new ways to make life safe, healthy, fun and interesting for the old. Think like a community, a brotherhood, not like atomized competing individuals a few of whom can somehow "beat the system" of the universe. Maybe it is better to examine clearly what we are with a view to understanding and acceptance than it is to try to escape what perhaps should be our inevitable ending.
death mortality cryogenics alcor geriatrics life singularity mind-uploading ray-kurzweil
CGA in 1024 Colors - a New Mode: the Illustrated Guide
awesome hackery. brings me back to my C=64 demo days
-
'a secret management and distribution service [from Square] that is now available for everyone. Keywhiz helps us with infrastructure secrets, including TLS certificates and keys, GPG keyrings, symmetric keys, database credentials, API tokens, and SSH keys for external services — and even some non-secrets like TLS trust stores. Automation with Keywhiz allows us to seamlessly distribute and generate the necessary secrets for our services, which provides a consistent and secure environment, and ultimately helps us ship faster. [...] Keywhiz has been extremely useful to Square. It's supported both widespread internal use of cryptography and a dynamic microservice architecture. Initially, Keywhiz use decoupled many amalgamations of configuration from secret content, which made secrets more secure and configuration more accessible. Over time, improvements have led to engineers not even realizing Keywhiz is there. It just works. Please check it out.'
square security ops keys pki key-distribution key-rotation fuse linux deployment secrets keywhiz
Bigcommerce Status Page blasts IBM Softlayer Object Storage service
This is pretty heavy stuff:
Bigcommerce engineers have been very pro-active in working with our storage provider, IBM Softlayer, in finding solutions. Unfortunately, it takes two parties to come to a solution. In this case, IBM Softlayer intentionally let their Object Storage cluster fall into disrepair and chose not to scale it. This has impacted Bigcommerce, IBM and many other Softlayer customers. Our engineers placed too much trust in IBM Softlayer and that's on us. However, the catastrophic failures to see metrics and rapidly scale capacity, the decisions to let hard drives sit at 90% utilization for weeks and months, the cascading failures of an undersized cluster of 52 nodes for the busiest data center in their business speaks to IBM Softlayer's lack of concern for their customers. We found this out 3 days ago.
(via Oisin)softlayer bigcommerce outages shambles ibm fail object-storage storage iaas cloud
Subscribing AWS Lambda Function To SNS Topic With aws-cli
how to use the AWS command line tools to do this
Yelp Product & Engineering Blog | True Zero Downtime HAProxy Reloads
Using tc and qdisc to delay SYNs while haproxy restarts. Definitely feels like on-host NAT between 2 haproxy processes would be cleaner and easier though!
linux networking hacks yelp haproxy uptime reliability tcp tc qdisc ops
-
Upsides of this new AWS service: * great UI and visualisations. * solid choice of metric to evaluate the results. Maybe things moved on since I was working on it, but the use of AUC, false positives and false negatives was pretty new when I was working on it. (er, 10 years ago!) Downsides: * it could do with more support for unsupervised learning algorithms. Supervised learning means you need to provide training data, which in itself can be hard work. My experience with logistic regression in the past is that it requires very accurate training data, too -- its tolerance for misclassified training examples is poor. * Also, in my experience, 80% of the hard work of using ML algorithms is writing good tokenisation and feature extraction algorithms. I don't see any help for that here unfortunately. (probably not that surprising as it requires really detailed knowledge of the input data to know what classes can be abbreviated into a single class, etc.)
Rob Pike's 5 rules of optimization
these are great. I've run into rule #3 ("fancy algorithms are slow when n is small, and n is usually small") several times...
twitter rob-pike via:igrigorik coding rules laws optimization performance algorithms data-structures aphorisms
AWS Lambda Event-Driven Architecture With Amazon SNS
Any message posted to an SNS topic can trigger the execution of custom code you have written, but you don't have to maintain any infrastructure to keep that code available to listen for those events and you don't have to pay for any infrastructure when the code is not being run. This is, in my opinion, the first time that Amazon can truly say that AWS Lambda is event-driven, as we now have a central, independent, event management system (SNS) where any authorized entity can trigger the event (post a message to a topic) and any authorized AWS Lambda function can listen for the event, and neither has to know about the other.
aws ec2 lambda sns events cep event-processing coding cloud hacks eric-hammond
Texting at the wheel kills more US teenagers every year than drink-driving
Texting while behind the wheel has overtaken drink driving as the biggest cause of death among teenagers in America. More than 3,000 teenagers are killed every year in car crashes caused by texting while driving compared to 2,700 from drink driving. The study by Cohen Children's Medical Center also discovered that 50 per cent of students admit to texting while driving.
-
Conducting such a widespread attack clearly demonstrates the weaponization of the Chinese Internet to co-opt arbitrary computers across the web and outside of China to achieve China's policy ends. The repurposing of the devices of unwitting users in foreign jurisdictions for covert attacks in the interests of one country's national priorities is a dangerous precedent — contrary to international norms and in violation of widespread domestic laws prohibiting the unauthorized use of computing and networked systems.
censorship ddos internet security china great-cannon citizen-lab reports web
-
How to build an Intelligent Personal Assistant: 'Sirius is an open end-to-end standalone speech and vision based intelligent personal assistant (IPA) similar to Apple's Siri, Google's Google Now, Microsoft's Cortana, and Amazon's Echo. Sirius implements the core functionalities of an IPA including speech recognition, image matching, natural language processing and a question-and-answer system. Sirius is developed by Clarity Lab at the University of Michigan. Sirius is published at the International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) 2015.'
sirius siri cortana google-now echo ok-google ipa assistants search video audio speech papers clarity nlp wikipedia
Why We Will Not Be Registering easyDNS.SUCKS - blog.easydns.org
If you're not immersed in the naming business you may find the jargon in it hard to understand. The basic upshot is this: the IPC believes that the mechanisms that were enacted to protect trademark holders during the deluge of new TLD rollouts are being gamed by the .SUCKS TLD operator to extort inflated fees from trademark holders.
(via Nelson)shakedown business internet domains dns easydns dot-sucks scams tlds trademarks ip
Data privacy is as important as tax, Google exec warns Noonan - Independent.ie
Yep, that would be Google requesting more regulation ;)
Russia just made a ton of Internet memes illegal - The Washington Post
In post-Soviet Russia, you don't make memes. Memes make (or unmake?) you. That is, at least, the only conclusion we can draw from an announcement made this week by Russia's three-year-old media agency/Internet censor Roskomnadzor, which made it illegal to publish any Internet meme that depicts a public figure in a way that has nothing to do with his "personality."
memes photoshop russia freedom web internet funny humour roskomnadzor censorship sad-keanu
-
'Utilities that help bridge the gap between Java 8 and Google Guava. Guava has the {@link FluentIterable} concept which is similar to streams. In many ways, fluent iterable is nicer, because it directly binds to the immutable collection classes. However, on balance it seems wise to use the stream API rather than {@code FluentIterable} in Java 8.'
-
I like the sound of this -- automated Java CMS GC tuning, kind of like a free version of JClarity's Censum (via Miguel Ángel Pastor)
J. G. Ballard predicted social media in a 1977 essay for Vogue
'In the intro essay to High Rise it says that J G Ballard predicted social media in a 1977 essay for Vogue. Here it is'
j-g-ballard social-media twitter instagram youtube future society vogue 1977 facebook media
Hacked French network exposed its own passwords during TV interview
lols
passwords post-its fail tv5monde authentication security tv funny
RADStack - an open source Lambda Architecture built on Druid, Kafka and Samza
'In this paper we presented the RADStack, a collection of complementary technologies that can be used together to power interactive analytic applications. The key pieces of the stack are Kafka, Samza, Hadoop, and Druid. Druid is designed for exploratory analytics and is optimized for low latency data exploration, aggregation, and ingestion, and is well suited for OLAP workflows. Samza and Hadoop complement Druid and add data processing functionality, and Kafka enables high throughput event delivery.'
druid samza kafka streaming cep lambda-architecture architecture hadoop big-data olap
-
an asynchronous Netty based graphite proxy. It protects Graphite from the herds of clients by minimizing context switches and interrupts; by batching and aggregating metrics. Gruffalo also allows you to replicate metrics between Graphite installations for DR scenarios, for example. Gruffalo can easily handle a massive amount of traffic, and thus increase your metrics delivery system availability. At Outbrain, we currently handle over 1700 concurrent connections, and over 2M metrics per minute per instance.
graphite backpressure metrics outbrain netty proxies gruffalo ops
Privacy Security Talk in TOG – 22nd April @ 7pm – FREE
Dublin is lucky enough to have great speakers pass through town on occasion and on Wednesday the 22nd April 2015, Runa A. Sandvik (@runasand) and Per Thorsheim (@thorsheim) have kindly offered to speak in TOG from 7pm. The format for the evening is a general meet and greet, but both speakers have offered to give a presentation on a topic of their choice. Anyone one interested in privacy, security, journalism, Tor and/or has previously attended a CryptoParty would be wise to attend. Doors are from 7pm and bring any projects with you you would like to share with other attendees. This is a free event, open to the public and no need to book. See you Wednesday. Runa A. Sandvik is an independent privacy and security researcher, working at the intersection of technology, law and policy. She contributes to The Tor Project, writes for Forbes, and is a technical advisor to both the Freedom of the Press Foundation and the TrueCrypt Audit project. Per Thorsheim as founder/organizer of PasswordsCon.org, his topic of choice is of course passwords, but in a much bigger context than most people imagine. Passwords, pins, biometrics, 2-factor authentication, security/usability and all the way into surveillance and protecting your health, kids and life itself.
privacy security runa-sandvik per-thorsheim passwords tor truecrypt tog via:oisin events dublin
-
'NSW officials seemed more interested in protecting their reputations than the integrity of elections. They sharply criticized Halderman and Teague, rather than commending them, for their discovery of the FREAK attack vulnerability. The Chief Information Officer of the Electoral Commission, Ian Brightwell, claimed Halderman and Teague's discovery was part of efforts by "well-funded, well-managed anti-internet voting lobby groups," an apparent reference to our friends at VerifiedVoting.org, where Halderman and Teague are voluntary Advisory Board members.1 Yet at the same time, Brightwell concluded that it was indeed possible that votes were manipulated.'
freak security vulnerabilities exploits nsw australia internet-voting vvat voting online-voting eff
Sheets of Glass Cut into Layered Ocean Waves by Ben Young
I particularly love "Rough Waters" -- amazing stuff from this kiwi artist
Working Time, Knowledge Work and Post-Industrial Society: Unpredictable Work - Aileen O'Carroll
my friend Aileen has written a book -- looks interesting:
I will argue that a key feature of working time within high-tech industries is unpredictability, which alters the way time is experienced and perceived. It affects all aspects of time, from working hours to work organisation, to career, to the distinction between work and life. Although many desire variety in work and the ability to control working hours, unpredictability causes dissatisfaction.
On Amazon.co.uk at: http://www.amazon.co.uk/Working-Time-Knowledge-Post-Industrial-Society-ebook/dp/B00VILIN4Ubooks reading time work society tech working-hours job life sociology
Introducing Vector: Netflix's On-Host Performance Monitoring Tool
It gives pinpoint real-time performance metric visibility to engineers working on specific hosts -- basically sending back system-level performance data to their browser, where a client-side renderer turns it into a usable dashboard. Essentially the idea is to replace having to ssh onto instances, run "top", systat, iostat, and so on.
vector netflix performance monitoring sysstat top iostat netstat metrics ops dashboards real-time linux
When S3's eventual consistency is REALLY eventual
a consistency outage in S3 last year, resulting in about 40 objects failing read-after-write consistency for a duration of about 23 hours
s3 eventual-consistency aws consistency read-after-writes bugs outages stackdriver
What is maximum Amazon S3 replication time on file upload? - Stack Overflow
Netflix note a 7 hour consistency delay
netflix aws s3 consistency eventual-consistency bugs outages
S3's "s3-external-1.amazonaws.com" endpoint
public documentation of how to work around the legacy S3 multi-region replication behaviour in North America
aws s3 eventual-consistency consistency us-east replication workarounds legacy
A collection of links for streaming algorithms and data structures
Good link-list from Debasish Ghosh
algorithms streaming big-data streams hll probabilistic data-structures frequency counting sketches cuckoo-filters bloom-filters minhash count-min
(SEC307) Building a DDoS-Resilient Architecture with AWS
good slides on a "web application firewall" proxy service, deployable as an auto-scaling EC2 unit
ec2 aws ddos security resilience slides reinvent firewalls http elb
Germanwings flight 4U9525: what's it like to listen to a black box recording?
After every air disaster, finding the black box recorder becomes the first priority – but for the crash investigators who have to listen to the tapes of people's final moments, the experience can be incredibly harrowing.
flight disasters metrics recording germanwings air-travel black-box-recorder flight-data-recorder death
Small claims triumph as aerial photographer routs flagrant infringers
This is great news. Flagrant copyright infringement of an aerial photograph penalised to the order of UKP 2,716
copyright infringement small-claims law uk webb-aviation photography images
Bad data PR: how the NSPCC sunk to a new low in data churnalism
when the NSPCC sent out a press release saying that one in ten 12-13 year olds [in the UK] are worried that they are addicted to porn and 12% have participated in sexually explicit videos, dozens of journalists appear to have simply played along – despite there being no report and little explanation of where the figures came from. [....] "It turns out the study was conducted by a "creative market research" [ie. pay-per-survey] group calledOnePoll. "Generate content and news angles with a OnePoll PR survey, and secure exposure for your brand," reads the company's blurb. "Our PR survey team can help draft questions, find news angles, design infographics, write and distribute your story." "The OnePoll survey included just 11 multiple-choice questions, which could be filled in online. Children were recruited via their parents, who were already signed up to OnePoll."
The NSPCC spends 25 million UKP per year on "child protection advice and awareness", so they have the money to do this right. Disappointing.nspcc bad-science bad-data methodology surveys porn uk kids addiction onepoll pr market-research
Stack Overflow Developer Survey 2015
wow, 52.5% of developers prefer a dark IDE theme?!
Gil Tene's "usual suspects" to reduce system-level hiccups/latency jitters in a Linux system
Based on empirical evidence (across many tens of sites thus far) and note-comparing with others, I use a list of "usual suspects" that I blame whenever they are not set to my liking and system-level hiccups are detected. Getting these settings right from the start often saves a bunch of playing around (and no, there is no "priority" to this - you should set them all right before looking for more advice...).
performance latency hiccups gil-tene tuning mechanical-sympathy hyperthreading linux ops
-
I think that materiality means what it says, and if people or algorithms do dumb things with trivial information that's their problem. But markets are a lot faster and more literal than they were when the materiality standard was created, and I wonder whether regulators or courts will one day decide that materiality is too reasonable a standard for modern markets. The materiality standard depends on the reasonable investor, and in many important contexts the reasonable investor has been replaced by a computer.
algorithms trading stock stock-market sec materiality april-fools-day tesla investing jokes
Time Series Metrics with Cassandra
slides from Chris Maxwell of Ubiquiti Networks describing what he had to do to get cyanite on Cassandra handling 30k metrics per second; an experimental "Date-tiered compaction" mode from Spotify was essential from the sounds of it. Very complex :(
cassandra spotify date-tiered-compaction metrics graphite cyanite chris-maxwell time-series-data
-
you can use 2-liter carbonated drink bottles to build an inexpensive, reusable water rocket. The thrill factor is surprisingly high, and you can fly them all day long for the cost of a little air and water. It's the perfect thing for those times when you just want to head down to the local soccer field and shoot off some rockets!
Outages, PostMortems, and Human Error 101
Good basic pres from John Allspaw, covering the basics of tier-one tech incident response -- defining the 5 severity levels; root cause analysis techniques (to Five-Whys or not); and the importance of service metrics
devops monitoring ops five-whys allspaw slides etsy codeascraft incident-response incidents severity root-cause postmortems outages reliability techops tier-one-support
Twitter's new anti-harassment filter
Twitter is calling it a "quality filter," and it's been rolling out to verified users running Twitter's iOS app since last week. It appears to work much like a spam filter, except instead of hiding bots and copy-paste marketers, it screens "threats, offensive language, [and] duplicate content" out of your notifications feed.
via Nelson5% of Google visitors have ad-injecting malware installed
Ad injectors were detected on all operating systems (Mac and Windows), and web browsers (Chrome, Firefox, IE) that were included in our test. More than 5% of people visiting Google sites have at least one ad injector installed. Within that group, half have at least two injectors installed and nearly one-third have at least four installed.
via Nelson.-
The horrors of monkey-patching:
I call out the Honeybadger gem specifically because was the most recent time I'd been bit by a seemingly good thing promoted in the community: monkey patching third party code. Now I don't fault Honeybadger for making their product this way. It provides their customers with direct business value: "just require 'honeybadger' and you're done!" I don't agree with this sort of practice. [....] I distrust everything [in Ruby] but a small set of libraries I've personally vetted or are authored by people I respect. Why is this important? Without a certain level of scrutiny you will introduce odd and hard to reproduce bugs. This is especially important because Ruby offers you absolutely zero guarantee whatever the state your program is when a given method is dispatched. Constants are not constants. Methods can be redefined at run time. Someone could have written a time sensitive monkey patch to randomly undefined methods from anything in ObjectSpace because they can. This example is so horribly bad that no one should every do, but the programming language allows this. Much worse, this code be arbitrarily inject by some transitive dependency (do you even know what yours are?).
ruby monkey-patching coding reliability bugs dependencies libraries honeybadger sinatra
Science is in crisis and scientists have lost confidence in Government policy
Excellent op-ed from Dr David McConnell, fellow emeritus of TCD's Smurfit Institute of Genetics: 'Ireland should once again foster, by competition, a good number of experienced, reputable people, of all ages, who have ideas about solving major scientific questions. These people are an essential part of the foundation of our science-based economy and society. Too many of them are no longer eligible for funding by SFI; too few are being appointed by the universities; and fewer PhDs are being awarded. The writing is on the wall.'
Salutin' Putin: inside a Russian troll house | World news | The Guardian
file under grim meathook future
grim-meathook-future guardian russia trolls social-media media censorship livejournal ideology social-control
-
As a result of a joint investigation of the events surrounding this incident by Google and CNNIC, we have decided that the CNNIC Root and EV CAs will no longer be recognized in Google products.
Llamasoft 8-bit game images now available for download
legal! go Jeff Minter
jeff-minter llamasoft yaks games history c=64 commodore vic-20 emulation via:shane
Cassandra remote code execution hole (CVE-2015-0225)
Ah now lads.
Under its default configuration, Cassandra binds an unauthenticated JMX/RMI interface to all network interfaces. As RMI is an API for the transport and remote execution of serialized Java, anyone with access to this interface can execute arbitrary code as the running user.
The Definitive Guide to the Music of The Big Lebowski | LA Weekly
definitive! (via Shero)
via:shero music the-big-lebowski la-weekly the-dude movies soundtracks
Reactive Programming for a demanding world
"building event-driven and responsive applications with RxJava", slides by Mario Fusco. Good info on practical Rx usage in Java
Chinese authorities compromise millions in cyberattacks
"[The] Great Firewall [of China] has switched from being a passive, inbound filter to being an active and aggressive outbound one."
china great-firewall censorship cyberwarfare github ddos baidu future
Avro, mail # dev - bytes and fixed handling in Python implementation - 2014-09-04, 22:54
More Avro trouble with "bytes" fields! Avoid using "bytes" fields in Avro if you plan to interoperate with either of the Python implementations; they both fail to marshal them into JSON format correctly. This is the official "avro" library, which produces UTF-8 errors when a non-UTF-8 byte is encountered
tebeka / fastavro / issues / #11 - fastavro breaks dumping binary fixed [4] — Bitbucket
The Python "fastavro" library cannot correctly render "bytes" fields. This is a bug, and the maintainer is acting in a really crappy manner in this thread. Avoid this library
fastavro fail bugs utf-8 bytes encoding asshats open-source python
A Team of Biohackers Has Figured Out How to Inject Your Eyeballs With Night Vision
Did it work? Yes. It started with shapes, hung about 10 meters away. "I'm talking like the size of my hand," Licina says. Before long, they were able to do longer distances, recognizing symbols and identifying moving subjects against different backgrounds. "The other test, we had people go stand in the woods," he says. "At 50 meters, we could figure out where they were, even if they were standing up against a tree." Each time, Licina had a 100% success rate. The control group, without being dosed with Ce6, only got them right a third of the time.
Well, that's some risky biohacking. wowbiohacking scary night-vision eyes chlorin-e6 infravision sfm
Tim Bray on one year as an xoogler
Seems pretty insightful; particularly "I do think the Internet economy would be better and more humane if it didn't have a single white-hot highly-overprivileged center. Also, sooner or later that'll stop scaling. Can't happen too soon."
google tim-bray via:nelson xoogler funding tech privacy ads internet
How I doubled my Internet speed with OpenWRT
File under "silly network hacks":
Comcast has an initiative called Xfinity WiFi. When you rent a cable modem/router combo from Comcast (as one of my nearby neighbors apparently does), in addition to broadcasting your own WiFi network, it is kind enough to also broadcast "xfinitywifi," a second "hotspot" network metered separately from your own.
By using his Buffalo WZR-HP-AG300H router's extra radio, he can load-balance across both his own paid-for connection, and the XFinity WiFi free one. ;)comcast diy networking openwrt routing home-network hacks xfinity-wifi buffalo
Unlocking the Power of Stable Teams with Twitter's SVP of Engineering - First Round Review
Huh. we do this in Swrve -- we call them "feature teams"
feature-team culture development teams coding twitter work teamwork
How We Scale VividCortex's Backend Systems - High Scalability
Excellent post from Baron Schwartz about their large-scale, 1-second-granularity time series database storage system
time-series tsd storage mysql sql baron-schwartz ops performance scalability scaling go
-
if (creation && object of art && algorithm && one's own algorithm) { include * an algorist * } elseif (!creation || !object of art || !algorithm || !one's own algorithm) { exclude * not an algorist * }
-
Race conditions, and errors at startup, seem to be particularly problematic
race-conditions startup bugs failure fault-tolerance hbase redis reliability ops papers concurrency exception-handling cassandra hdfs mapreduce
You Cannot Have Exactly-Once Delivery
Cut out and keep:
Within the context of a distributed system, you cannot have exactly-once message delivery. Web browser and server? Distributed. Server and database? Distributed. Server and message queue? Distributed. You cannot have exactly-once delivery semantics in any of these situations.
distributed distcomp exactly-once-delivery networking outages network-partitions byzantine-generals reference
What's confusing about Kafka: a list
At a recent call, Neha said "The most confusing behavior we have is how producing to a topic can return errors for few seconds after the topic was already created". As she said that, I remembered that indeed, this was once very confusing, but then I got used to it. Which got us thinking: What other things that Kafka does are very confusing to new users, but we got so used to them that we no longer even see the issue?
-
This is the second part of our guide on streaming data and Apache Kafka. In part one I talked about the uses for real-time data streams and explained our idea of a stream data platform. The remainder of this guide will contain specific advice on how to go about building a stream data platform in your organization.
tl;dr: limit the number of Kafka clusters; use Avro.architecture kafka storage streaming event-processing avro schema confluent best-practices tips
The Four Month Bug: JVM statistics cause garbage collection pauses (evanjones.ca)
Ugh, tying GC safepoints to disk I/O? bad idea:
The JVM by default exports statistics by mmap-ing a file in /tmp (hsperfdata). On Linux, modifying a mmap-ed file can block until disk I/O completes, which can be hundreds of milliseconds. Since the JVM modifies these statistics during garbage collection and safepoints, this causes pauses that are hundreds of milliseconds long. To reduce worst-case pause latencies, add the -XX:+PerfDisableSharedMem JVM flag to disable this feature. This will break tools that read this file, like jstat.
Gradle Team Perspective on Bazel
interesting.
(SDD401) Amazon Elastic MapReduce Deep Dive and Best Practices
good slides for EMR tuning from re:Invent 2014
-
LOL. grepping commit logs for /bug|fix/ does the job, apparently:
In the literature, Rahman et al. found that a very cheap algorithm actually performs almost as well as some very expensive bug-prediction algorithms. They found that simply ranking files by the number of times they've been changed with a bug-fixing commit (i.e. a commit which fixes a bug) will find the hot spots in a code base. Simple! This matches our intuition: if a file keeps requiring bug-fixes, it must be a hot spot because developers are clearly struggling with it.
bugs rahman-algorithm heuristics source-code-analysis coding algorithms google static-code-analysis version-control
Build in the Cloud: Accessing Source Code
Google reinvented ClearCase
Cross-Region Replication for Amazon S3
Amazing it took so long
ECJ case debates EU citizens' right to privacy
The US wields secretive and indiscriminate powers to collect data, he said, and had never offered Brussels any commitments to guarantee EU privacy standards for its citizens' data. On the contrary, said [Max Schrems' counsel] Mr Hoffmann, "Safe Harbour" provisions could be overruled by US domestic law at any time. Thus he asked the court for a full judicial review of the "illegal" Safe Harbour principles which, he said, violated the essence of privacy and left EU citizens "effectively stripped of any protection". [Irish] DPC counsel Paul Anthony McDermott SC suggested that Mr Schrems had not been harmed in any way by the status quo. "This is not surprising, given that the NSA isn't currently interested in the essays of law students in Austria," he said. Mr Travers for Mr Schrems disagreed, saying "the breach of the right to privacy is itself the harm".
ireland dpc data-protection privacy eu ec ecj law rights safe-harbour
EU-US data pact skewered in court hearing
A lawyer for the European Commission told an EU judge on Tuesday (24 March) he should close his Facebook page if he wants to stop the US snooping on him, in what amounts to an admission that Safe Harbour, an EU-US data protection pact, doesn't work.
safe-harbour privacy data-protection ecj eu ec surveillance facebook nsa gchq
devbook/README.md at master · barsoom/devbook
How to avoid the shitty behaviour of ActiveRecord wrt migration safety, particularly around removing/renaming columns. ugh, ActiveRecord
activerecord fail rails mysql sql migrations databases schemas releasing
Papa's Maze 2.0: a father's beautifully intricate puzzle for his daughter
Working in a similar fashion – drawing small portions each day – it took Mr. Nomura about 2 months to complete his new maze. And in our humble opinion, we think it's actually just as beautiful, if not more. It's not quite as dense and the crisper lines make it easier to perceive the interesting patterns that the maze forms. It's stunning in graphic quality but it's also a functioning solvable maze, just like its predecessor. Say hello to Papa's Maze 2.0. It's available as a print for $30.
The official REST Proxy for Kafka
The REST Proxy is an open source HTTP-based proxy for your Kafka cluster. The API supports many interactions with your cluster, including producing and consuming messages and accessing cluster metadata such as the set of topics and mapping of partitions to brokers. Just as with Kafka, it can work with arbitrary binary data, but also includes first-class support for Avro and integrates well with Confluent's Schema Registry. And it is scalable, designed to be deployed in clusters and work with a variety of load balancing solutions. We built the REST Proxy first and foremost to meet the growing demands of many organizations that want to use Kafka, but also want more freedom to select languages beyond those for which stable native clients exist today. However, it also includes functionality beyond traditional clients, making it useful for building tools for managing your Kafka cluster. See the documentation for a more detailed description of the included features.
kafka rest proxies http confluent queues messaging streams architecture
-
'Caffeine is a Java 8 based concurrency library that provides specialized data structures, such as a high performance cache.'
cache java8 java guava caching concurrency data-structures coding
Combining static model checking with dynamic enforcement using the Statecall Policy Language
This looks quite nice -- a model-checker "for regular programmers". Example model for ping(1):
01 automaton ping (int max_count, int count, bool can_timeout) { 02 Initialize; 03 during { 04 count = 0; 05 do { 06 Transmit_Ping; 07 either { 08 Receive_Ping; 09 } or (can_timeout) { 10 Timeout_Ping; 11 }; 12 count = count + 1; 13 } until (count >= max_count); 14 } handle { 15 SIGINFO; 16 Print_Summary; 17 };ping model-checking models formal-methods verification static dynamic coding debugging testing distcomp papers
-
good review
cdt replication distcomp voldemort dynamo riak storage papers
-
Google open sources a key part of their internal build system (internally called "Blaze" it seems for a while). Very nice indeed!
blaze bazel build-tools building open-source google coding packaging
-
a Nix-based continuous build system, released under the terms of the GNU GPLv3 or (at your option) any later version. It continuously checks out sources of software projects from version management systems to build, test and release them. The build tasks are described using Nix expressions. This allows a Hydra build task to specify all the dependencies needed to build or test a project. It supports a number of operating systems, such as various GNU/Linux flavours, Mac OS X, and Windows.
-
"tees" all TCP traffic from one server to another. "widely used by companies in China"!
testing benchmarking performance tcp ip tcpcopy tee china regression-testing stress-testing ops
Managing private Nix packages outside the Nixpkgs tree
Useful for private-repo Nix usage
Top 10 AWS Security Best Practices: #6 - Rotate all the Keys Regularly
Good doc on how to perform key rotation in AWS
[Nix-dev] Pulling a programs source code from a git repo
Nix supports building from git sha. excellent
Transparent huge pages implicated in Redis OOM
A nasty real-world prod error scenario worsened by THPs:
jemalloc(3) extensively uses madvise(2) to notify the operating system that it's done with a range of memory which it had previously malloc'ed. The page size on this machine is 2MB because transparent huge pages are in use. As such, a lot of the memory which is being marked with madvise(..., MADV_DONTNEED) is within substantially smaller ranges than 2MB. This means that the operating system never was able to evict pages which had ranges marked as MADV_DONTNEED because the entire page has to be unneeded to allow a page to be reused. Despite initially looking like a leak, the operating system itself was unable to free memory because of madvise(2) and transparent huge pages. This led to sustained memory pressure on the machine and redis-server eventually getting OOM killed.
oom-killer oom linux ops thp jemalloc huge-pages madvise redis memory
AllCrypt hacked, via PHP, Wordpress, and the marketing director's email
critical flaw: gaining access to the MySQL db let the attacker manipulate account balances. oh dear
-
'inspires kids to explore and learn about science, engineering, and technology—and have fun doing it. Every month, a new crate to help kids develop a tinkering mindset and creative problem solving skills.' aimed at ages 9-14+
kids gifts tinkering stem education fun engineering science toys
-
Some nice performance tricks; I particularly like the use of sljit:
Ag uses Pthreads to take advantage of multiple CPU cores and search files in parallel. Files are mmap()ed instead of read into a buffer. Literal string searching uses Boyer-Moore strstr. Regex searching uses PCRE's JIT compiler (if Ag is built with PCRE >=8.21). Ag calls pcre_study() before executing the same regex on every file. Instead of calling fnmatch() on every pattern in your ignore files, non-regex patterns are loaded into arrays and binary searched.
jit cli grep search ack ag unix pcre sljit boyer-moore tools
Richard Stallman's GNU Manifesto Turns Thirty
nice New Yorker profile of rms
-
Thought-provoking article looking back to John Perry Barlow's "A Declaration of the Independence of Cyberspace", published in 1996:
Barlow once wrote that "trusting the government with your privacy is like having a Peeping Tom install your window blinds." But the Barlovian focus on government overreach leaves its author and other libertarians blind to the same encroachments on our autonomy from the private sector. The bold and romantic techno-utopian ideals of "A Declaration" no longer need to be fought for, because they're already gone.
john-perry-barlow 1990s history cyberspace internet surveillance privacy data-protection libertarianism utopian manifestos
The Terrible Technical Interview
TechCrunch, very down on the traditional big-O-and-whiteboard tech interview. See also https://news.ycombinator.com/item?id=9243169 for some good comments at HN. To be honest I think a good comprehension of data structures and big-O is pretty vital though....
The myopia boom seems to be due to spending too much time indoors
via Tony Finch
eyes health neuroscience science vision nature myopia short-sightedness
-
Some neat new features for Mark Fletcher's mailing-lists-as-a-service site: Markdown support, manageable archives (GREAT feature!), subgroups, calendars, files and wiki.
wiki email mailman mailing-lists mlm markdown mark-fletcher groups.io collaboration
Stairs to nowhere, trap streets, and other Toronto oddities
'There's a set of stairs on Greenwood Avenue that lead nowhere. At the top, a wooden fence at the end of someone's back yard blocks any further movement, forcing the climber to turn around and descend back to the street. What's remarkable about the pointless Greenwood stairs, which were built in 1959 as a shortcut to a now-demolished brickyard, is that someone still routinely maintains them: in winter, some kindly soul deposits a scattering of salt lest one of the stairs' phantom users slip; in summer someone comes with a broom to sweep away leaves. These urban leftovers are lovingly called "Thomassons" after Gary Thomasson, a former slugger for the San Francisco Giants, Oakland As, Yankees, Dodgers, and, most fatefully, the Yomiuri Giants in Tokyo.'
trap-streets maps ip google via:bldgblog mapping copyright thomassons orphaned-roads
President's message gets lost in (automated) translation
In a series of bizarre translations, YouTube's automated translation service took artistic licence with the [President's] words of warmth. When the head of state sent St Patrick's Day greetings to viewers, the video sharing site said US comedian Tina Fey was being "particular with me head". As President Higgins spoke of his admiration for Irish emigrants starting new communities abroad, YouTube said the President referenced blackjack and how he "just couldn't put the new iPhone" down. And, in perhaps the most unusual moment, as he talked of people whose hearts have sympathy, the President "explained" he was once on a show "that will bar a gift card".
(via Daragh O'Brien)lol president ireland michael-d-higgins automation translation machine-learning via:daraghobrien funny blackjack iphone tina-fey st-patrick fail
Irish government under fire for turning its back on basic research : Nature News & Comment
Pretty much ALL of Ireland's research scientists have put their names to an open letter to the Irish government, decrying the state of science funding, published this week in "Nature". 'Although total spending on research and development grew through the recession, helped by foreign investments, Ireland's government has cut state spending on research (see 'Celtic tiger tamed'). It also prioritized grants in 14 narrow areas — ones in which either large global markets exist, or in which Irish companies are competitive. These include marine renewable energy, smart grids, medical devices and computing. The effect has been to asphyxiate the many areas of fundamental science — including astrophysics, particle physics and areas of the life sciences — that have been deprived of funding, several researchers in Ireland told Nature. "The current policies are having a very significant detrimental effect on the health and viability of the Irish scientific ecosystem," says Kevin Mitchell, a geneticist who studies the basis of neurological disorders at Trinity College Dublin. "Research that cannot be shoehorned into one of the 14 prioritized areas has been ineligible for most funding," he says.' That's another fine mess Sean Sherlock has gotten us into :(
sean-sherlock fail ireland research government funding grants science tcd kevin-mitchell life-sciences nature
Mars One finalist Dr. Joseph Roche rips into the project
So, here are the facts as we understand them: Mars One has almost no money. Mars One has no contracts with private aerospace suppliers who are building technology for future deep-space missions. Mars One has no TV production partner. Mars One has no publicly known investment partnerships with major brands. Mars One has no plans for a training facility where its candidates would prepare themselves. Mars One's candidates have been vetted by a single person, in a 10-minute Skype interview. "My nightmare about it is that people continue to support it and give it money and attention, and it then gets to the point where it inevitably falls on its face," said Roche. If, as a result, "people lose faith in NASA and possibly even in scientists, then that's the polar opposite of what I'm about. If I was somehow linked to something that could do damage to the public perception of science, that is my nightmare scenario."
science space mars-one tcd joseph-roche nasa mars exploration scams
Stu Hood and Brian Degenhardt, Scala at Twitter, SF Scala @Twitter 20150217
'Stu Hood and Brian Degenhardt talk about the history of Scala at Twitter, from inception until today, covering 2.10 migration, the original Alex Payne's presentation from way back, pants, and more. The first five years of Scala at Twitter and the years ahead!' Very positive indeed on the monorepo concept.
monorepo talks scala sfscala stu-hood twitter pants history repos build projects compilation gradle maven sbt
demonstration of the importance of server-side request timeouts
from MongoDB, but similar issues often apply in many other TCP/HTTP-based systems
-
an open source stream processing software system developed by Mozilla. Heka is a "Swiss Army Knife" type tool for data processing, useful for a wide variety of different tasks, such as: Loading and parsing log files from a file system. Accepting statsd type metrics data for aggregation and forwarding to upstream time series data stores such as graphite or InfluxDB. Launching external processes to gather operational data from the local system. Performing real time analysis, graphing, and anomaly detection on any data flowing through the Heka pipeline. Shipping data from one location to another via the use of an external transport (such as AMQP) or directly (via TCP). Delivering processed data to one or more persistent data stores.
Via feylya on twitter. Looks potentially niftyheka mozilla monitoring metrics via:feylya ops statsd graphite stream-processing
Real World Crypto 2015: Password Hashing according to Facebook
Very interesting walkthrough of how Facebook hash user passwords, including years of accreted practices
-
My account got hacked, running up over $600 in charges. Here's the conclusion after running through the Sony support gauntlet. They can only refund up to $150. I can dispute the charges with my bank, but that will result in my account being banned. I cannot unban my account, and will thus lose my purchases ("but you only have the Last of Us and some of our free games, so it's not a big deal") Whomever hacked my account deactivated my PS4, and activated their own. Customer support will only permit one activation every 6 months. I'm locked out of logging into my own account on my PS4 for six months.
games sony psn playstation fail ps4 hacking security customer-support horror-stories
Goodbye MongoDB, Hello PostgreSQL
Another core problem we've faced is one of the fundamental features of MongoDB (or any other schemaless storage engine): the lack of a schema. The lack of a schema may sound interesting, and in some cases it can certainly have its benefits. However, for many the usage of a schemaless storage engine leads to the problem of implicit schemas. These schemas aren't defined by your storage engine but instead are defined based on application behaviour and expectations.
Well, don't say we didn't warn you ;)mongodb mysql postgresql databases storage schemas war-stories
Apple Appstore STATUS_CODE_ERROR causes worldwide service problems
Particularly notable for this horrific misfeature, noted by jgc:
I can't commit code at CloudFlare because we use two-factor auth for the VPN (and everything else) and non-Apple apps on my iPhone are asking for my iTunes password. Tried airplane mode and apps simply don't load at all!
That is a _disastrous_ policy choice by Apple. Does this mean Apple can shut down third-party app operation on iOS devices worldwide should they feel like it?2fa authy apps ios apple ownership itunes outages appstore fail jgc
Correcting YCSB's Coordinated Omission problem
excellent walkthrough of CO and how it affects Yahoo!'s Cloud Storage Benchmarking platform
coordinated-omission co yahoo ycsb benchmarks performance testing
Backblaze Vaults: Zettabyte-Scale Cloud Storage Architecture
Backblaze deliver their take on nearline storage: 'Backblaze's cloud storage Vaults deliver 99.99999% annual durability, horizontal scalability, and 20 Gbps of per-Vault performance, while being operationally efficient and extremely cost effective. Driven from the same mindset that we brought to the storage market with Backblaze Storage Pods, Backblaze Vaults continue our singular focus of building the most cost-efficient cloud storage around.'
architecture backup storage backblaze nearline offline reed-solomon error-correction
Ireland accused of weakening data rules
Privacy campaign group Lobbyplag puts Ireland one of top three offenders in pushing for changes to EU privacy law
privacy data-protection lobbyplag ireland eu germany lobbying
-
the stock-photo counterpart to "Women Eating Salad" has been found
Can Spark Streaming survive Chaos Monkey?
good empirical results on Spark's resilience to network/host outages in EC2
ec2 aws emr spark resilience ha fault-tolerance chaos-monkey netflix
-
Concourse is a CI system composed of simple tools and ideas. It can express entire pipelines, integrating with arbitrary resources, or it can be used to execute one-off builds, either locally or in another CI system.
ci concourse-ci build deployment continuous-integration continuous-deployment devops
Epsilon Interactive breach the Fukushima of the Email Industry (CAUCE)
Upon gaining access to an ESP, the criminals then steal subscriber data (PII such as names, addresses, telephone numbers and email addresses, and in one case, Vehicle Identification Numbers). They then use ESPs' mailing facility to send spam; to monetize their illicit acquisition, the criminals have spammed ads for fake Adobe Acrobat and Skype software. On March 30, the Epsilon Interactive division of Alliance Data Marketing (ADS on NASDAQ) suffered a massive breach that upped the ante, substantially. Email lists of at least eight financial institutions were stolen. Thus far, puzzlingly, Epsilon has refused to release the names of compromised clients. [...] The obvious issue at hand is the ability of the thieves to now undertake targeted spear-phishing problem as critically serious as it could possibly be.
cauce epsilon-interactive esp email pii data-protection spear-phishing phishing identity-theft security ads
In Ukraine, Tomorrow's Drone War Is Alive Today
Drones, hackerspaces and crowdfunding:
The most sophisticated UAV that has come out of the Ukrainian side since the start of the conflict is called the PD-1 from developer Igor Korolenko. It has a wingspan of nearly 10 feet, a five-hour flight time, carries electro-optical and infrared sensors as well as a video camera that broadcasts on a 128 bit encrypted channel. Its most important feature is the autopilot software that allows the drone to return home in the event that the global positioning system link is jammed or lost. Drone-based intelligence gathering is often depicted as risk-free compared to manned aircraft or human intelligence gathering, but, says Korolenko, if the drone isn't secure or the signature is too obvious, the human coasts can be very, very high. "Russian military sometimes track locations of ground control stations," he wrote Defense One in an email. "Therefore UAV squads have to follow certain security measures - to relocate frequently, to move out antennas and work from shelter, etc. As far as I know, two members of UAV squads were killed from mortar attacks after [their] positions were tracked by Russian electronic warfare equipment."
(via bldgblog)via:bldgblog war drones uav future ukraine russia tech aircraft pd-1 crowdfunding
-
a 303 and an 808 in your browser. this is deadly
Ubuntu To Officially Switch To systemd Next Monday - Slashdot
Jesus. This is going to be the biggest shitfest in the history of Linux...
-
A project to reduce systemd to a base initd, process supervisor and transactional dependency system, while minimizing intrusiveness and isolationism. Basically, it's systemd with the superfluous stuff cut out, a (relatively) coherent idea of what it wants to be, support for non-glibc platforms and an approach that aims to minimize complicated design. uselessd is still in its early stages and it is not recommended for regular use or system integration.
This may be the best option to evade the horrors of systemd. Japan's Robot Dogs Get Funerals as Sony Looks Away
in July 2014, [Sony's] repairs [of Aibo robot dogs] stopped and owners were left to look elsewhere for help. The Sony stiff has led not only to the formation of support groups--where Aibo enthusiasts can share tips and help each other with repairs--but has fed the bionic pet vet industry. "The people who have them feel their presence and personality," Nobuyuki Narimatsu, director of A-Fun, a repair company for robot dogs, told AFP. "So we think that somehow, they really have souls." While concerted repair efforts have kept many an Aibo alive, a shortage of spare parts means that some of their lives have come to an end.
sony aibo robots japan dogs pets weird future badiotday iot gadgets
"Cuckoo Filter: Practically Better Than Bloom"
'We propose a new data structure called the cuckoo filter that can replace Bloom filters for approximate set membership tests. Cuckoo filters support adding and removing items dynamically while achieving even higher performance than Bloom filters. For applications that store many items and target moderately low false positive rates, cuckoo filters have lower space overhead than space-optimized Bloom filters. Our experimental results also show that cuckoo filters outperform previous data structures that extend Bloom filters to support deletions substantially in both time and space.'
algorithms paper bloom-filters cuckoo-filters cuckoo-hashing data-structures false-positives big-data probabilistic hashing set-membership approximation
Amazing cutting from Vanity Fair, 1896, for International Women's Day
"The sisters make a pretty picture on the platform ; but it is not women of their type who need to assert themselves over Man. However, it amuses them--and others ; and I doubt if the tyrant has much to fear from their little arrows." Constance Markievicz was one of those sisters, and the other was Eva Gore-Booth.
markievicz history ireland sligo vanity-fair 19th-century dismissal sexism iwd women
-
Authy doesn't come off well here: 'Authy should have been harder to break. It's an app, like Authenticator, and it never left Davis' phone. But Eve simply reset the app on her phone using a mail.com address and a new confirmation code, again sent by a voice call. A few minutes after 3AM, the Authy account moved under Eve's control.'
authy security hacking mfa authentication google apps exploits
Ask the Decoder: Did I sign up for a global sleep study?
How meaningful is this corporate data science, anyway? Given the tech-savvy people in the Bay Area, Jawbone likely had a very dense sample of Jawbone wearers to draw from for its Napa earthquake analysis. That allowed it to look at proximity to the epicenter of the earthquake from location information. Jawbone boasts its sample population of roughly "1 million Up wearers who track their sleep using Up by Jawbone." But when looking into patterns county by county in the U.S., Jawbone states, it takes certain statistical liberties to show granularity while accounting for places where there may not be many Jawbone users. So while Jawbone data can show us interesting things about sleep patterns across a very large population, we have to remember how selective that population is. Jawbone wearers are people who can afford a $129 wearable fitness gadget and the smartphone or computer to interact with the output from the device. Jawbone is sharing what it learns with the public, but think of all the public health interests or other third parties that might be interested in other research questions from a large scale data set. Yet this data is not collected with scientific processes and controls and is not treated with the rigor and scrutiny that a scientific study requires. Jawbone and other fitness trackers don't give us the option to use their devices while opting out of contributing to the anonymous data sets they publish. Maybe that ought to change.
jawbone privacy data-protection anonymization aggregation data medicine health earthquakes statistics iot wearables
Pinterest's highly-available configuration service
Stored on S3, update notifications pushed to clients via Zookeeper
A Journey into Microservices | Hailo Tech Blog
Excellent three-parter from Hailo, describing their RabbitMQ+Go-based microservices architecture. Very impressive!
-
The Large Hadron Migrator is a tool to perform live database migrations in a Rails app without locking.
The basic idea is to perform the migration online while the system is live, without locking the table. In contrast to OAK and the facebook tool, we only use a copy table and triggers. The Large Hadron is a test driven Ruby solution which can easily be dropped into an ActiveRecord or DataMapper migration. It presumes a single auto incremented numerical primary key called id as per the Rails convention. Unlike the twitter solution, it does not require the presence of an indexed updated_at column.
migrations database sql ops mysql rails ruby lhm soundcloud activerecord
Biased Locking in HotSpot (David Dice's Weblog)
This is pretty nuts. If biased locking in the HotSpot JVM is causing performance issues, it can be turned off:
You can avoid biased locking on a per-object basis by calling System.identityHashCode(o). If the object is already biased, assigning an identity hashCode will result in revocation, otherwise, the assignment of a hashCode() will make the object ineligible for subsequent biased locking.
hashcode jvm java biased-locking locking mutex synchronization locks performance
A Zero-Administration Amazon Redshift Database Loader - AWS Big Data Blog
nifty!
Archie Markup Language (ArchieML)
ArchieML (or "AML") was created at The New York Times to make it easier to write and edit structured text on deadline that could be rendered in web pages, or more specifically, rendered in interactive graphics. One of the main goals was to make it easy to tag text as data, without having type a lot of special characters. Another goal was to allow the document to contain lots of notes and draft text that would not be read into the data. And finally, because we make extensive use of Google Documents's concurrent-editing features — while working on a graphic, we can have several reporters, editors and developers all pouring information into a single document — we wanted to have a format that could survive being edited by users who may never have seen ArchieML or any other markup language at all before.
California Says Motorcycle Lane-Splitting Is Hella Safe
A recent yearlong study by the California Office of Traffic Safety has found motorcycle lane-splitting to be a safe practice on public roads. The study looked at collisions involving 7836 motorcyclists reported by 80 police departments between August 2012 and August 2013. "What we learned is, if you lane-split in a safe or prudent manner, it is no more dangerous than motorcycling in any other circumstance," state spokesman Chris Cochran told the Sacramento Bee. "If you are speeding or have a wide speed differential (with other traffic), that is where the fatalities came about."
lane-splitting cycling motorcycling bikes road-safety driving safety california
-
Good terminology for this concept:
The try server runs a similar configuration to the continuous integration server, except that it is triggered not on commits but on "try job request", in order to test code pre-commit.
See also https://wiki.mozilla.org/ReleaseEngineering/TryServer for the Moz take on it.build ci integration try-server jenkins buildbot chromium development
-
A Dropwizard Metrics extension to instrument JDBC resources and measure SQL execution times.
HP is trying to patent Continuous Delivery
This is appalling bollocks from HP:
On 1st March 2015 I discovered that in 2012 HP had filed a patent (WO2014027990) with the USPO for 'Performance tests in a continuous deployment pipeline' (the patent was granted in 2014). [....] HP has filed several patents covering standard Continuous Delivery (CD) practices. You can help to have these patents revoked by providing 'prior art' examples on Stack Exchange.
In fairness, though, this kind of shit happens in most big tech companies. This is what happens when you have a broken software patenting system, with big rewards for companies who obtain shitty troll patents like these, and in turn have companies who reward the engineers who sell themselves out to write up concepts which they know have prior art. Software patents are broken by design!cd devops hp continuous-deployment testing deployment performance patents swpats prior-art
Exponential Backoff And Jitter
Great go-to explainer blog post for this key distributed-systems reliability concept, from the always-solid Marc Brooker
marc-brooker distsys networking backoff exponential jitter retrying retries reliability occ
17 Things Everyone Must Eat In Dublin
actually a fairly sane list of lunchy options -- the SMS fish finger butty is a lunch staple for us Swrvers
VividCortex uses K-Means Clustering to discover related metrics
After selecting an interesting spike in a metric, the algorithm can automate picking out a selection of other metrics which spiked at the same time. I can see that being pretty damn useful
metrics k-means-clustering clustering algorithms discovery similarity vividcortex analysis data
Alibaba's cloud service launches in US, wants to rain all over Amazon
server-hosting only for now. Interesting!
Alibaba's cloud platform already competes with the likes of AWS in China. Aliyun's Chinese data centers are in Beijing, Hangzhou, Qingdao, Hong Kong, and Shenzhen. "For the time being, we are just testing the water," Yu said today. That means Aliyun will focus first on Chinese companies doing business in the US. "We know well what Chinese clients need, and now it's time for us to learn what US clients need," he added.
-
the following guidelines maximize bandwidth usage: Optimizing the sizes of the file parts, whether they are part of a large file or an entire small file; Optimizing the number of parts transferred concurrently. Tuning these two parameters achieves the best possible transfer speeds to [S3].
-
Excellent web-based ASCII-art editor (via Craig)
via:craig design ascii diagrams editor ascii-art art asciiflow drawing
Services Engineering Reading List
good list of papers/articles for fans of scalability etc.
architecture papers reading reliability scalability articles to-read
-
nice, free-during-beta Mac app to draw ASCII-art diagrams
-
"Open source APM for Java" -- profiling in production, with a demo benchmark showing about a 2% performance impact. Wonder about effects on memory/GC, though
"Everything you've ever said to Siri/Cortana has been recorded...and I get to listen to it"
This should be a reminder.
At first, I though these sound bites were completely random. Then I began to notice a pattern. Soon, I realized that I was hearing peoples commands given to their mobile devices. Guys, I'm telling you, if you've said it to your phone, it's been recorded...and there's a damn good chance a 3rd party is going to hear it.
privacy google siri cortana android voice-recognition outsourcing mobile
-
Fantastic 1997-era book of interviews with the programmers behind some of the greatest games in retrogaming history:
Halcyon Days: Interviews with Classic Computer and Video Game Programmers was released as a commercial product in March 1997. At the time it was one of the first retrogaming projects to focus on lost history rather than game collecting, and certainly the first entirely devoted to the game authors themselves. Now a good number of the interviewees have their own web sites, but none of them did when I started contacting them in 1995. [...] If you have any of the giddy anticipation that I did whenever I picked up a magazine containing an interview with Mark Turmell or Dan [M.U.L.E.] Bunten, then you want to start reading.
Pub Table Quiz - In Aid of Digital Rights Ireland
Jason Roe is organising a Table Quiz in Dublin on March 26th to support fundraising efforts by Digital Rights Ireland. We will supply tables, questions and a ready supply of beer and maybe finger food.
Why are transhumanists such dicks?
Good discussion from a transhumanist forum (via Boing Boing):
"I've been around and interviewed quite a lot of self-identified transhumanists in the last couple of years, and I've noticed many of them express a fairly stark ideology that is at best libertarian, and at worst Randian. Very much "I want super bionic limbs and screw the rest of the world". They tend to brush aside the ethical, environmental, social and political ramifications of human augmentation so long as they get to have their toys. There's also a common expression that if sections of society are harmed by transhumanist progress, then it is unfortunate but necessary for the greater good (the greater good often being bestowed primarily upon those endorsing the transhumanism). That attitude isn't prevalent on this forum at all – I think the site tends to attract more practical body-modders than theoretical transhumanists – but I wondered if anyone else here had experienced the same attitudes in their own circles? What do you make of it?"
transhumanism evolution body-modding surgery philosophy via:boingboing libertarianism society politics
Release Protocol Buffers v3.0.0-alpha-2 · google/protobuf
New major-version track for protobuf, with some interesting new features: Removal of field presence logic for primitive value fields, removal of required fields, and removal of default values. This makes proto3 significantly easier to implement with open struct representations, as in languages like Android Java, Objective C, or Go. Removal of unknown fields. Removal of extensions, which are instead replaced by a new standard type called Any. Fix semantics for unknown enum values. Addition of maps. Addition of a small set of standard types for representation of time, dynamic data, etc. A well-defined encoding in JSON as an alternative to binary proto encoding.
protobuf binary marshalling serialization google grpc proto3 coding open-source
RIPQ: Advanced photo caching on flash for Facebook
Interesting priority-queue algorithm optimised for caching data on SSD
priority-queue algorithms facebook ssd flash caching ripq papers
-
Performance-diagnosis-as-a-service. Cool.
Users download and install an Illuminate Daemon using a simple installer which starts up a small stand alone Java process. The Daemon sits quietly unless it is asked to start gathering SLA data and/or to trigger a diagnosis. Users can set SLA's via the dashboard and can opt to collect latency measurements of their transactions manually (using our library) or by asking Illuminate to automatically instrument their code (Servlet and JDBC based transactions are currently supported). SLA latency data for transactions is collected on a short cycle. When the moving average of latency measurements goes above the SLA value (e.g. 150ms), a diagnosis is triggered. The diagnosis is very quick, gathering key data from O/S, JVM(s), virtualisation and other areas of the system. The data is then run through the machine learned algorithm which will quickly narrow down the possible causes and gather a little extra data if needed. Once Illuminate has determined the root cause of the performance problem, the diagnosis report is sent back to the dashboard and an alert is sent to the user. That alert contains a link to the result of the diagnosis which the user can share with colleagues. Illuminate has all sorts of backoff strategies to ensure that users don't get too many alerts of the same type in rapid succession!
illuminate jclarity java jvm scala latency gc tuning performance
-
Binary message marshalling, client/server stubs generated by an IDL compiler, bidirectional binary protocol. CORBA is back from the dead! Intro blog post: http://googledevelopers.blogspot.ie/2015/02/introducing-grpc-new-open-source-http2.html Relevant: Steve Vinoski's commentary on protobuf-rpc back in 2008: http://steve.vinoski.net/blog/2008/07/13/protocol-buffers-leaky-rpc/
Bloom Cookies: web search personalization without user tracking
Interesting paper
bloom-cookies bloom-filters data-structures cookies privacy personalization user-tracking http
Why we run an open source program - Walmart Labs
This is a great exposition of why it's in a company's interest to engage with open source. Not sure I agree with 'engineers are the artists of our generation' but the rest are spot on
-
MQTT definitely has a smaller size on the wire. It's also simpler to parse (let's face it, Huffman isn't that easy to implement) and provides guaranteed delivery to cater to shaky wireless networks. On the other hand, it's also not terribly extensible. There aren't a whole lot of headers and options available, and there's no way to make custom ones without touching the payload of the message. It seems that HTTP/2 could definitely serve as a reasonable replacement for MQTT. It's reasonably small, supports multiple paradigms (pub/sub & request/response) and is extensible. Its also supported by the IETF (whereas MQTT is hosted by OASIS). From conversations I've had with industry leaders in the embedded software and chip manufacturing, they only want to support standards from the IETF. Many of them are still planning to support MQTT, but they're not happy about it. I think MQTT is better at many of the things it was designed for, but I'm interested to see over time if those advantages are enough to outweigh the benefits of HTTP. Regardless, MQTT has been gaining a lot of traction in the past year or two, so you may be forced into using it while HTTP/2 catches up.
Automatically Deploy from GitHub Using AWS CodeDeploy - Application Management Blog
I like this
Programmer IS A Career Path, Thank You
Well said -- Amazon had a good story around this btw
how Curator fixed issues with the Hive ZooKeeper Lock Manager Implementation
Ugh, ZK is a bear to work with.
Apache Curator is open source software which is able to handle all of the above scenarios transparently. Curator is a Netflix ZooKeeper Library and it provides a high-level API, CuratorFramework, that simplifies using ZooKeeper. By using a singleton CuratorFramework instance in the new ZooKeeperHiveLockManager implementation, we not only fixed the ZooKeeper connection issues, but also made the code easy to understand and maintain.
zookeeper apis curator netflix distributed-locks coding hive
Advanced cryptographic ratcheting
Forward secrecy and in-session key "ratcheting"
crypto privacy key-management forward-secrecy pfs key-ratcheting key-rotation
-
What a mess.
What's faster: PV, HVM, HVM with PV drivers, PVHVM, or PVH? Cloud computing providers using Xen can offer different virtualization "modes", based on paravirtualization (PV), hardware virtual machine (HVM), or a hybrid of them. As a customer, you may be required to choose one of these. So, which one?
Proving that Android's, Java's and Python's sorting algorithm is broken (and showing how to fix it)
Wow, this is excellent work. A formal verification of Tim Peters' TimSort failed, resulting in a bugfix:
While attempting to verify TimSort, we failed to establish its instance invariant. Analysing the reason, we discovered a bug in TimSort's implementation leading to an ArrayOutOfBoundsException for certain inputs. We suggested a proper fix for the culprit method (without losing measurable performance) and we have formally proven that the fix actually is correct and that this bug no longer persists.
timsort algorithms android java python sorting formal-methods proofs openjdk
-
"Cheap SSL certs from $4.99/yr" -- apparently recommended for cheap, low-end SSL certs
-
Erasure codes, such as Reed-Solomon (RS) codes, are increasingly being deployed as an alternative to data-replication for fault tolerance in distributed storage systems. While RS codes provide significant savings in storage space, they can impose a huge burden on the I/O and network resources when reconstructing failed or otherwise unavailable data. A recent class of erasure codes, called minimum-storage-regeneration (MSR) codes, has emerged as a superior alternative to the popular RS codes, in that it minimizes network transfers during reconstruction while also being optimal with respect to storage and reliability. However, existing practical MSR codes do not address the increasingly important problem of I/O overhead incurred during reconstructions, and are, in general, inferior to RS codes in this regard. In this paper, we design erasure codes that are simultaneously optimal in terms of I/O, storage, and network bandwidth. Our design builds on top of a class of powerful practical codes, called the product-matrix-MSR codes. Evaluations show that our proposed design results in a significant reduction the number of I/Os consumed during reconstructions (a 5 reduction for typical parameters), while retaining optimality with respect to storage, reliability, and network bandwidth.
erasure-coding reed-solomon compression reliability reconstruction replication fault-tolerance storage bandwidth usenix papers
Everyday I'm Shuffling - Tips for Writing Better Spark Programs [slides]
Two Spark experts from Databricks provide some good tips
Cowen went golfing and officials dithered as country burned in 2008 - Independent.ie
Lest we forget, the sheer bullshitting ineptitude of Fianna Fail as they managed to shamble into destroying Ireland's economy in 2008:
Once that nasty bit of business was done, the Cabinet departed en masse for six weeks on their summer holidays, despite the emerging economic and financial tsunami. Cowen and family famously took up residence in a caravan park in Connemara as opposed to his 'official' residence at the Mannin Bay Hotel nearby. When pressed by our reporter Niamh Horan as to why he was not at his station, he defensively replied: "I don't understand it. First the media have a go at me because I'm taking a holiday with my family and then they come down to see if I'm having a good time!" he exclaimed.
2008 meltdown ireland brian-cowen connemara politics history fianna-fail
How I Became A Minor Celebrity In China (After My Stolen Phone Ended Up There)
Phone is stolen, shipped to China, and winds up being bought by "Brother Orange" -- then the story becomes China's biggest viral hit
-
40 minutes of multi-zone network outage for majority of instances. 'The internal software system which programs GCE's virtual network for VM egress traffic stopped issuing updated routing information. The cause of this interruption is still under active investigation. Cached route information provided a defense in depth against missing updates, but GCE VM egress traffic started to be dropped as the cached routes expired.' I wonder if Google Pimms fired the alarms for this ;)
Listen to a song made from data lost during MP3 conversion
Ryan McGuire, a PhD student in Composition and Computer Technologies at the University of Virginia Center for Computer Music, has created the project The Ghost In The MP3 [....] For his first trick, McGuire took Suzanne Vega's 'Tom's Diner' and drained it into a vaporous piece titled 'moDernisT." McGuire chose the track he explains on his site because it was famously used as one of the main controls in the listening tests used to develop the MP3 algorithm.
-
A gateway script, now included in PCP
-
System performance metrics framework, plugged by Netflix, open-source for ages
open-source pcp performance system metrics ops red-hat netflix
Superfish: A History Of Malware Complaints And International Surveillance - Forbes
Superfish, founded and led by former Intel employee and ex-surveillance boffin Adi Pinhas, has been criticised by users the world over since its inception in 2006.
superfish lenovo privacy surveillance ads java windows mac firefox pups ssl tls ad-injection komodia
The Superfish certificate has been cracked, exposing Lenovo users to attack | The Verge
The cracked certificate exposes Lenovo users to man-in-the-middle attacks, similar to those opened up by Heartbleed. Armed with this password and the right software, a coffee shop owner could potentially spy on any Lenovo user on her network, collecting any passwords that were entered during the session. The evil barista could also insert malware into the data stream at will, disguised as a software update or a trusted site.
Amazingly stupid.Police have asked Dropcam for video from people's home cameras -- Fusion
"Like any responsible father, Hugh Morrison had installed cameras in every room in the flat," is the opening line of Intrusion, a 2012 novel set in the near future. Originally installed so that Hugh and his wife can keep an eye on their kids, the Internet-connected cameras wind up being used later in the novel by police who tap into the feeds to monitor the couple chatting on their couch when they are suspected of anti-societal behavior. As with so many sci-fi scenarios, the novel's vision was prophetic. People are increasingly putting small Internet-connected cameras into their homes. And law enforcement officials are using the cameras to collect evidence about them.
Extracting the SuperFish certificate
not exactly the most challenging reverse I've ever seen ;)
reverse-engineering security crypto hacking tls ssl superfish lenovo
The Great SIM Heist: How Spies Stole the Keys to the Encryption Castle
Holy shit. Gemalto totally rooted.
With [Gemalto's] stolen encryption keys, intelligence agencies can monitor mobile communications without seeking or receiving approval from telecom companies and foreign governments. Possessing the keys also sidesteps the need to get a warrant or a wiretap, while leaving no trace on the wireless provider's network that the communications were intercepted. Bulk key theft additionally enables the intelligence agencies to unlock any previously encrypted communications they had already intercepted, but did not yet have the ability to decrypt. [...] According to one secret GCHQ slide, the British intelligence agency penetrated Gemalto's internal networks, planting malware on several computers, giving GCHQ secret access. We "believe we have their entire network," the slide's author boasted about the operation against Gemalto.
encryption security crypto nsa gchq gemalto smartcards sim-cards privacy surveillance spying
One year of InfluxDB and the road to 1.0
half of the [Monitorama] attendees were employees and entrepreneurs at monitoring, metrics, DevOps, and server analytics companies. Most of them had a story about how their metrics API was their key intellectual property that took them years to develop. The other half of the attendees were developers at larger organizations that were rolling their own DevOps stack from a collection of open source tools. Almost all of them were creating a "time series database" with a bunch of web services code on top of some other database or just using Graphite. When everyone is repeating the same work, it's not key intellectual property or a differentiator, it's a barrier to entry. Not only that, it's something that is hindering innovation in this space since everyone has to spend their first year or two getting to the point where they can start building something real. It's like building a web company in 1998. You have to spend millions of dollars and a year building infrastructure, racking servers, and getting everything ready before you could run the application. Monitoring and analytics applications should not be like this.
graphite monitoring metrics tsd time-series analytics influxdb open-source
-
Sysdig Cloud users have the ability to view and analyze Java Management Extensions (JMX) metrics out of the box with no additional configuration or setup required.
Will the madness never end? Komodia SSL certificates are EVERYWHERE
I think that at this point it is safe to assume that any SSL interception product sold by Komodia or based on the Komodia SDK is going to be using the same method. What does this mean? Well, this means that those dodgy certificates aren't limited to Lenovo laptops sold over a specific date range. It means that anyone who has come into contact with a Komodia product, or who has had some sort of Parental Control software installed on their computer should probably check to see if they are affected.
Twitter's Answers architecture
Twitter's mobile-device analytics service architecture, with Kafka and Storm in full Lambda-Architecture mode
-
The argument for the "monorepo" -- ie. lots of projects in a single Git repo. There's lots more discussion pro/con on twitter, e.g.: https://twitter.com/search?q=monorepo&src=typd , https://twitter.com/hivetheory/timelines/449385567982067713
-
Poor hardware imaging practices, basically:
It looks like all devices with the fingerprint are Dropbear SSH instances that have been deployed by Telefonica de Espana. It appears that some of their networking equipment comes setup with SSH by default, and the manufacturer decided to re-use the same operating system image across all devices.
FreeBSD breaks its kernel RNG for 4 months
If you are running a current kernel r273872 or later, please upgrade your kernel to r278907 or later immediately and regenerate keys. I discovered an issue where the new framework code was not calling randomdev_init_reader, which means that read_random(9) was not returning good random data. This means most/all keys generated may be predictable and must be regenerated.
What every programmer should know about solid-state drives
Lots of good advice here for dealing with SSDs
Azul Zing on Ubuntu on AWS Marketplace
hmmm, very interesting -- the super-low-latency Zing JVM is available as a commercial EC2 instance type, at costs less than the EC2 instance price
Crowdsourcing isn't broken — Backchannel — Medium
'A great compendium by @harper of techniques for handling trolls and griefers in online communities', via kragen
via:kragen antispam filtering trolls community crowdsourcing threadless harper griefers abuse tips
South Korea faces $1bn bill after hackers raid national ID database • The Register
Simon McGarr says: '80% of S.Korea's population have had their ID number stolen, crimewave ongoing. >> Turns out a pot of honey is sweet'
fail south-korea korea security id-cards ssn id-numbers privacy
Sign up for Privacy International's anti-surveillance campaign
Have you ever made a phone call, sent an email, or, you know, used the internet? Of course you have! Chances are, at some point over the past decade, your communications were swept up by the U.S. National Security Agency. The NSA then shares information with the UK Government's intelligence agency GCHQ by default. A recent court ruling found that this sharing was unlawful. But no one could find out if their records were collected and then illegally shared between these two agencies… until now! Because of our recent victory against the UK intelligence agency in court, now anyone in the world — yes, ANYONE, including you — can find out if GCHQ illegally received information about you from the NSA. Join our campaign by entering your details below to find out if GCHQ illegally spied on you, and confirm via the email we send you. We'll then go to court demanding that they finally come clean on unlawful surveillance.
gchq nsa spying surveillance internet phone uk law campaign privacy-international
How "omnipotent" hackers tied to NSA hid for 14 years—and were found at last
'"Equation Group" ran the most advanced hacking operation ever uncovered.' Mad stuff. The security industry totally failed here
nsa privacy security surveillance hacking keyloggers malware
-
decent set of intro slides
Apache Spark: A Delight for Developers | Cloudera Engineering Blog
Another Spark intro blog post
-
'JOL (Java Object Layout) is the tiny toolbox to analyze object layout schemes in JVMs. These tools are using Unsafe, JVMTI, and Serviceability Agent (SA) heavily to decoder the actual object layout, footprint, and references. This makes JOL much more accurate than other tools relying on heap dumps, specification assumptions, etc.' Recommended by Nitsan Wakart, looks pretty useful for JVM devs
java jvm tools scala memory estimation ram object-layout debugging via:nitsan
HdrHistogram: A better latency capture method
An excellent intro to HdrHistogram usage
hdrhistogram hdr histograms statistics latency measurement metrics percentiles quantiles gil-tene nitsan-wakart
-
Butterfield insists that Slack improves on the basic messaging functionality offered by its predecessors. The company plans to expand from 100 employees to 250 this year, open an office in Dublin, and launch a version that supports large companies with multiple teams.
-
A tool for managing Apache Kafka. It supports the following : Manage multiple clusters; Easy inspection of cluster state (topics, brokers, replica distribution, partition distribution); Run preferred replica election; Generate partition assignments (based on current state of cluster); Run reassignment of partition (based on generated assignments)
Vaurien, the Chaos TCP Proxy — Vaurien 1.8 documentation
Vaurien is basically a Chaos Monkey for your TCP connections. Vaurien acts as a proxy between your application and any backend. You can use it in your functional tests or even on a real deployment through the command-line. Vaurien is a TCP proxy that simply reads data sent to it and pass it to a backend, and vice-versa. It has built-in protocols: TCP, HTTP, Redis & Memcache. The TCP protocol is the default one and just sucks data on both sides and pass it along. Having higher-level protocols is mandatory in some cases, when Vaurien needs to read a specific amount of data in the sockets, or when you need to be aware of the kind of response you're waiting for, and so on. Vaurien also has behaviors. A behavior is a class that's going to be invoked everytime Vaurien proxies a request. That's how you can impact the behavior of the proxy. For instance, adding a delay or degrading the response can be implemented in a behavior. Both protocols and behaviors are plugins, allowing you to extend Vaurien by adding new ones. Last (but not least), Vaurien provides a couple of APIs you can use to change the behavior of the proxy live. That's handy when you are doing functional tests against your server: you can for instance start to add big delays and see how your web application reacts.
proxy tcp vaurien chaos-monkey testing functional-testing failures sockets redis memcache http
Embed-able Computers are a Thing. — February 12, 2015
'If it works, a copy of Burgertime for DOS is now in your browser, clickable from my entry. If it doesn't… well, no Burgertime for you. (Unless you visit the page.) There's a "share this" link in the new archive.org interface for sharing these in-browser emulations in web pages, weblogs and who knows what else.'
sharing embeds html javascript emulation msdos burgertime games archive.org
-
According to a report posted Thursday to the website of the state-run China Youth Daily, the Cyberspace Administration of China choral group this week unveiled a new song, "Cyberspace Spirit," glorifying the cleanliness and clarity of China's uniquely managed Internet. The song, an orchestral march built around a chorus that proclaims China's ambition to become an "Internet power," opens with lyrics describing celestial bodies keeping careful watch over the sky. From there, the lyrics conjure more vivid imagery, comparing the Internet to "a beam of incorruptible sunlight" that unites "the powers of life from all creation."
china great-firewall censorship music songs cyberspace-spirit omgwtfbbq
An Algorithm to Extract Looping GIFs From Videos
/r/loopinggifs, automated
Samsung's smart TVs are inserting unwanted ads into users' own movies
Amazingly shitty. Never buying a Samsung TV if this is what they think is acceptable
Massive thumbs-down Docker review
extensive.
The Pizza Party Where Everyone Got Fired
The testers at [MAJOR PUBLISHER] had just finished wrapping up testing on a project we'll call "Biolands." And to congratulate them, the man in charge arranged a huge bowling/pizza party for the end of the week. Of course everyone is hyped for the event. So the day finally arrives and all the testers show up. They all start bowling and eating pizza. After a few hours of everyone enjoying themselves, the VP asks for everyone's attention. When he does manage to get the team to listen, he begins to thank them for their hard work and has the leads hand them their termination papers.
And many other horror stories from the worst software industry of all -- games.games software jobs bowling pizza fired horror-stories hr employment
Automating Tinder with Eigenfaces
While my friends were getting sucked into "swiping" all day on their phones with Tinder, I eventually got fed up and designed a piece of software that automates everything on Tinder.
This is awesome. (via waxy)via:waxy tinder eigenfaces machine-learning k-nearest-neighbour algorithms automation ai
-
Our latest open source release from Swrve Labs: an Apache-licensed, SLF4J-compatible, simple, fluent API for rate-limited logging in Java: 'A RateLimitedLog object tracks the rate of log message emission, imposes an internal rate limit, and will efficiently suppress logging if this is exceeded. When a log is suppressed, at the end of the limit period, another log message is output indicating how many log lines were suppressed. This style of rate limiting is the same as the one used by UNIX syslog; this means it should be comprehensible, easy to predict, and familiar to many users, unlike more complex adaptive rate limits.' We've been using this in production for months -- it's pretty nifty ;) Never fear your logs again!
logs logging coding java open-source swrve slf4j rate-limiting libraries
BENCHMARKING THE RASPBERRY PI 2
Retro console emulation! Mario Kart and Ocarina of Time and Conker's Bad Fur Day! Nobody actually builds stuff with the Raspberry Pi, it's just an odd form of nostalgic consumerism wrapped up in a faddish 'making' trend! The original Raspberry Pi saw a lot of emulator use, but it was limited: the Pi 1 could handle the NES, SNES, Genesis/Mega Drive, and other earlier consoles with ease. Emulator performance for N64 and original Playstation games was just barely unplayable. Now, the Raspi 2 can easily handle N64 and PSX games. [HoZyVN] tried out N64's Mario Kart and PSX's Spyro the Dragon. They're playable, and an entire generation rushed out to Microcenter to relive their glory days of sitting with their faces embedded in a console television drinking Sunny D all day.
raspberry-pi emulation n64 playstation gaming hardware benchmarks
"Man vs Machine: Practical Adversarial Detection of Malicious Crowdsourcing Workers" [paper]
"traditional ML techniques are accurate (95%–99%) in detection but can be highly vulnerable to adversarial attacks". ain't that the truth
security adversarial-attacks machine-learning paper crowdsourcing via:kragen
-
Nice looking static code validation tool for Java, from Google. I recognise a few of these errors ;)
Totally Mexico! How the Nathan Barley nightmare came true | Television & radio | The Guardian
Nathan Barley was scarcely less prophetic when it came to TV itself. In one episode Nathan's friend Claire makes a comically po-faced, self-righteous but secretly rather narcissistic documentary about a choir made up of drug addicts. Nine years later, Channel 4 made Addicts' Symphony for real.
nathan-barley well-weapon vice shoreditch drugs charlie-brooker chris-morris sitcoms channel-4
Google Maps Tenth Anniversary | Re/code
the whole story of GMaps
0x74696d | Falling In And Out Of Love with DynamoDB, Part II
Good DynamoDB real-world experience post, via Mitch Garnaat. We should write up ours, although it's pretty scary-stuff-free by comparison
South Korean spymaster had a team posting political comments on Twitter and rigging polls
Mad stuff. The South Korean National Intelligence Service directly interfering in a democratic election by posting fake comments and rigging online polls
web polls twitter social-media psyops korea south-korea nis sock-puppets democracy
Dept of Education and Primary Online Database
Simon McGarr has a theory -- the indefinite data retention of sensitive data on primary schoolchildren actually has a genesis in the Irish state wishing to protect itself against prosecution from future child abuse cases
UK-US surveillance regime was unlawful 'for seven years' | UK news | The Guardian
The regime that governs the sharing between Britain and the US of electronic communications intercepted in bulk was unlawful until last year, a secretive UK tribunal has ruled. The Investigatory Powers Tribunal (IPT) declared on Friday that regulations covering access by Britain's GCHQ to emails and phone records intercepted by the US National Security Agency (NSA) breached human rights law.
Digital Rights Ireland announces its first conference!
Digital Rights Europe, Wednesday, April 15th in Dublin. deadly!
digital-rights ireland dri privacy data-protection europe eu
Twitter CEO: 'We suck at dealing with abuse' | The Verge
'We suck at dealing with abuse and trolls on the platform and we've sucked at it for years. It's no secret and the rest of the world talks about it every day. We lose core user after core user by not addressing simple trolling issues that they face every day. I'm frankly ashamed of how poorly we've dealt with this issue during my tenure as CEO. It's absurd. There's no excuse for it. I take full responsibility for not being more aggressive on this front. It's nobody else's fault but mine, and it's embarrassing. We're going to start kicking these people off right and left and making sure that when they issue their ridiculous attacks, nobody hears them. Everybody on the leadership team knows this is vital.'
More like this!-
nice deep-dive from Adrian Colyer
The DOs and DON'Ts of Blue/Green Deployment - CloudNative
Excellent post -- Delta sounds like a very well-designed product
Can we have medical privacy, cloud computing and genomics all at the same time?
Today sees the publication of a report I [Ross Anderson] helped to write for the Nuffield Bioethics Council on what happens to medical ethics in a world of cloud-based medical records and pervasive genomics. As the information we gave to our doctors in private to help them treat us is now collected and treated as an industrial raw material, there has been scandal after scandal. From failures of anonymisation through unethical sales to the care.data catastrophe, things just seem to get worse. Where is it all going, and what must a medical data user do to behave ethically? We put forward four principles. First, respect persons; do not treat their confidential data like were coal or bauxite. Second, respect established human-rights and data-protection law, rather than trying to find ways round it. Third, consult people who'll be affected or who have morally relevant interests. And fourth, tell them what you've done – including errors and security breaches.
ethics medicine health data care.data privacy healthcare ross-anderson genomics data-protection human-rights
Comparing Message Queue Architectures on AWS
A good overview -- I like the summary table. tl;dr:
If you are light on DevOps and not latency sensitive use SQS for job management and Kinesis for event stream processing. If latency is an issue, use ELB or 2 RabbitMQs (or 2 beanstalkds) for job management and Redis for event stream processing.
amazon architecture aws messaging queueing elb rabbitmq beanstalk kinesis sqs redis kafka
TL;DR: Cassandra Java Huge Pages
Al Tobey does some trial runs of -XX:+AlwaysPreTouch and -XX:+UseHugePages
Enjoy Bintray and use it as pain-free gateway to Maven Central
ahh, interesting! This looks much easier (via JBaruch)
-
Marc Brooker: 'When it comes to building working software in the long term, the emotional pursuit of craft is not as important as the human pursuit of teamwork, or the intellectual pursuit of correctness. Patterns is one of the most powerful ideas we have. The critics may be right that it devalues the craft, but we would all do well to remember that the craft of software is a means, not an end.'
-
Via Walter, the best description of the appeal of Minecraft I've read:
Minecraft is exceptionally good at intrinsic narrative. It recognises, preserves and rewards everything you do. It presses you to play frontiersman. A Minecraft world ends up dotted with torchlit paths, menhirs, landmarks, emergency caches. Here's the hole where you dug stone for your first house. Here's the causeway you built from your spawn point to a handy woodland. Here's the crater in the landscape where the exploding monster took out you and your wheatfield at once. And, of course, here's your enormous castle above a waterfall. There's no utility in building anything bigger than a hut, but the temptations of architecture are irresistible. Minecraft isn't so much a world generator as a screenshot-generator and a war-story generator. This is what will get the game the bulk of its critical attention, and deservedly so. That's why I want to call attention to the extrinsic narrative. It's minimal, implicit, accidental and very powerful. It's this: you wake alone beside an endless sea in a pristine, infinite wilderness. The world is yours. You can literally sculpt mountains, with time and effort. You'll die and be reborn on the beach where you woke first. You'll walk across the world forever and never see another face. You can build a whole empire of roads and palaces and beacon towers, and the population of that empire will only ever be you. When you leave, your towers will stand empty forever. I haven't seen that surfaced in a game before. It's strong wine.
Backstage Blog - Prometheus: Monitoring at SoundCloud - SoundCloud Developers
whoa, this is pretty excellent. The major improvement over a graphite-based system would be the multi-dimensional tagging of metrics, which we currently have to do by simply expanding the graphite metric's name to encompass all those dimensions and use searching at query time, inefficiently.
monitoring soundcloud prometheus metrics service-metrics graphite alerting
'Prometheus instrumentation library for JVM applications'
Good example of a clean java OSS release, from Soundcloud. will be copying bits of this myself soon...
prometheus java libraries oss github sonatype maven releases
-
A good set of basic, controversy-free guidelines for clean java code style
style java google coding guidelines formatting coding-standards
A Brief History of NSA Backdoors
from 1946 to present
nsa security backdoors sigint actel dual_ec_drbg crypto-ag crypto
Study: You Can't Change an Anti-Vaxxer's Mind
According to a major new study in the journal 'Pediatrics', trying to [persuade anti-vaxxers to vaccinate] may actually make the problem worse. The paper tested the effectiveness of four separate pro-vaccine messages, three of which were based very closely on how the Centers for Disease Control and Prevention (CDC) itself talks about vaccines. The results can only be called grim: Not a single one of the messages was successful when it came to increasing parents' professed intent to vaccinate their children. And in several cases the messages actually backfired, either increasing the ill-founded belief that vaccines cause autism or even, in one case, apparently reducing parents' intent to vaccinate.
vaccination health measles mmr autism facts via:mrneutron stupidity cdc papers vaccines
-
"dysaguria" is the perfect noun, and "dysagurian" is the perfect adjective, to describe the eponymous company in Dave Eggers' The Circle. It's not in the same league as Orwell, or Huxley, or Bradbury, or Burgess. But it does raise very important questions about what could possibly go wrong if one company controlled all the world's information. In the novel, the company operates according to the motto "all that happens must be known"; and one of its bosses, Eamon Bailey, encourages everywoman employee Mae Holland to live an always-on (clear, transparent) life according the maxims "secrets are lies", "sharing is caring", and "privacy is theft". Eggers's debts to dystopian fiction are apparent. But, whereas writers like Orwell, Huxley, Bradbury, and Burgess were concerned with totalitarian states, Eggers is concerned with a totalitarian company. However, the noun "dystopia" and the adjective "dystopian" – perfect though they are for the terror of military/security authoritarianism in 1984, or Brave new World, or Farenheit 451, or A Clockwork Orange – do not to my mind encapsulate the nightmare of industrial/corporate tyranny in The Circle. On the other hand, "dysaguria" as a noun and "dysagurian" as an adjective, in my view really do capture the essence of that "frightening company".
dysaguria dystopia future sf authoritarianism surveillance the-circle google facebook
-
Via negatendo: 'I would like to share my excitement about the fact that after almost a year of development, an instance of my NetHack bot has finally managed to ascend a game for the first time without human interventions, wizard mode cheats or bones stuffing, and did so at the public server at acehack.de.' The bot is written in Clojure. Apparently 'pudding farming' did the trick...
-
lol.
funny data-science statistics machine-learning hadoop bayes memes image-macros
NA Server Roadmap Update: PoPs, Peering, and the North Bridge
League of Legends has set up private network links to a variety of major US ISPs to avoid internet weather (via Nelson)
-
Because there exists no method known to man, more terribly suited to expose the cosmic meaningless of existence than pairing the words of H.P. Lovecraft with seemingly delightful and charming pictures of adorable kittens.
8 gdb tricks you should know (Ksplice Blog)
These are very good -- bookmarking for the next time I'm using gdb, probably about 3 years from now
EFF's Game Plan for Ending Global Mass Surveillance
For years, we've been working on a strategy to end mass surveillance of digital communications of innocent people worldwide. Today we're laying out the plan, so you can understand how all the pieces fit together—that is, how U.S. advocacy and policy efforts connect to the international fight and vice versa. Decide for yourself where you can get involved to make the biggest difference. This plan isn't for the next two weeks or three months. It's a multi-year battle that may need to be revised many times as we better understand the tools and authorities of entities engaged in mass surveillance and as more disclosures by whistleblowers help shine light on surveillance abuses.
-
This group aims to consolidate opposition, give clear information and support letter writing and information awareness against the Dept. of Education's Primary Online Database.
Apple Pay suffering fraud problems
Fraud in Apple Pay will in time, come to be managed – but the fact that easily available PII can waylay best in class protection should give us all pause.
Excellent example of failed "anonymisation" of a dataset
Fred Logue notes how this failed Mayo TD Michelle Mulherin:
From recent reports it mow appears that the Department of Education is discussing anonymisation of the Primary Online Database with the Data Protection Commissioner. Well someone should ask Mayo TD Michelle Mulherin how anonymisation is working for her. The Sunday Times reports that Ms Mulherin was the only TD in the Irish parliament on the dates when expensive phone calls were made to a mobile number in Kenya. The details of the calls were released under the Freedom of Information Act in an "anonymised" database. While it must be said the fact that Ms Mulherin was the only TD present on those occasions does not prove she made the calls – the reporting in the press is now raising the possibility that it was her. From a data protection point of view this is a perfect example of the difficulty with anonymisation. Data protection rules apply to personal data which is defined as data relating to a living individual who is or can be identified from the data or from the data in conjunction with other information. Anonymisation is often cited as a means for processing data outside the scope of data protection law but as Ms Mulherin has discovered individuals can be identified using supposedly anonymised data when analysed in conjunction with other data. In the case of the mysterious calls to Kenya even though the released information was "anonymised" to protect the privacy of public representatives, the phone log used in combination with the attendance record of public representatives and information on social media was sufficient to identify individuals and at least raise evidence of association between individuals and certain phone calls. While this may be well and good in terms of accounting for abuses of the phone service it also has worrying implications for the ability of public representatives to conduct their business in private. The bottom line is that anonymisation is very difficult if not impossible as Ms Mulherin has learned to her cost. It certainly is a lot more complex than simply removing names and other identifying features from a single dataset. The more data that there is and the more diverse the sources the greater the risk that individuals can be identified from supposedly anonymised datasets.
data anonymisation fred-logue ireland michelle-mulherin tds kenya data-protection privacy
Publishing from GitHub to Maven Central
A good starting point. This looks bloody complex :(
-
Nice wrapper for 'tc' and 'netem', for network latency/packet loss emulation
networking testing linux tc netem latency packet-loss iptables
Visualizing AWS Storage with Real-Time Latency Spectrogram
ohhhh this is very nice indeed. Great viz!
dataviz latency io ops sysdig charts graphs commandline linux
Stop Playing Monopoly With Your Kids (And Play These Games Instead) | FiveThirtyEight
538 apply their numbercrunching skills to the BoardGameGeek ratings index
ODROID-C1 - Multicore credit card computer
Pretty amazing specs for a 33 quid SBC.
Amlogic ARM® Cortex®-A5(ARMv7) 1.5Ghz quad core CPUs * Mali™-450 MP2 GPU (OpenGL ES 2.0/1.1 enabled for Linux and Android) * 1Gbyte DDR3 SDRAM * Gigabit Ethernet * 40pin GPIOs * eMMC4.5 HS200 Flash Storage slot / UHS-1 SDR50 MicroSD Card slot * USB 2.0 Host x 4, USB OTG x 1, * Infrared(IR) Receiver * Uses Ubuntu 14.04 or Android KitKat operating systems
Includes HDMI out. (via Conor O'Neill)How TCP backlog works in Linux
good description of the process
Swiss Authorities Arrest Bot for Buying Drugs and Fake Passport
A bot created by a group of artists spent the last few months selecting items at random from a Silk Road-style darknet marketplace, buying them with Bitcoin, and having them shipped to a gallery in Switzerland. After the it bought some ecstasy pills and a counterfeit passport, we asked: How will authorities deal with the complex legal and moral issue of a piece of artificial intelligence breaking the law? It turns out, the answer was simple: just arrest the computer.
drugs darknet bitcoin ecstasy art bots law-enforcement switzerland
-
Java Concurrency Tools for the JVM. This project aims to offer some concurrent data structures currently missing from the JDK: Bounded lock free queues SPSC/MPSC/SPMC/MPMC variations for concurrent queues Alternative interfaces for queues (experimental) Offheap concurrent ring buffer for ITC/IPC purposes (experimental) Executor (planned)
Functional Programming Patterns (BuildStuff '14)
Good, and very accessible even for FP noobs like myself ;)
How to reduce the JVM thread stack size
"-Xss" switch
Maintaining performance in distributed systems [slides]
Great slide deck from Elasticsearch on JVM/dist-sys performance optimization
-
Nice trick -- wrap servers with a libc wrapper to intercept bind(2) and accept(2) calls, so that transparent restarts becode possible
linux ops servers uptime restarting libc bind accept sockets
How to Catch a Terrorist - The New Yorker
This is spot on --
By flooding the system with false positives, big-data approaches to counterterrorism might actually make it harder to identify real terrorists before they act. Two years before the Boston Marathon bombing, Tamerlan Tsarnaev, the older of the two brothers alleged to have committed the attack, was assessed by the city's Joint Terrorism Task Force. They determined that he was not a threat. This was one of about a thousand assessments that the Boston J.T.T.F. conducted that year, a number that had nearly doubled in the previous two years, according to the Boston F.B.I. As of 2013, the Justice Department has trained nearly three hundred thousand law-enforcement officers in how to file "suspicious-activity reports." In 2010, a central database held about three thousand of these reports; by 2012 it had grown to almost twenty-eight thousand. "The bigger haystack makes it harder to find the needle," Sensenbrenner told me. Thomas Drake, a former N.S.A. executive and whistle-blower who has become one of the agency's most vocal critics, told me, "If you target everything, there's no target."
terrorism false-positives filtering detection jttf nsa fbi surveillance gchq
-
'All deleted tweets from politicians'. Great idea
delete twitter politics politicians ireland social-media news
Zoë Keating on getting a shitty deal from Google's new Music Key licensing
The Youtube music service was introduced to me as a win win and they don't understand why I don't see it that way. "We are trying to create a new revenue stream on top of the platform that exists today." A lot of people in the music industry talk about Google as evil. I don't think they are evil. I think they, like other tech companies, are just idealistic in a way that works best for them. I think this because I used to be one of them. The people who work at Google, Facebook, etc can't imagine how everything they make is not, like, totally awesome. If it's not awesome for you it's because you just don't understand it yet and you'll come around. They can't imagine scenarios outside their reality and that is how they inadvertently unleash things like the algorithmic cruelty of Facebook's yearly review (which showed me a picture I had posted after a doctor told me my husband had 6-8 weeks to live).
google business music youtube zoe-keating music-key licensing tech
-
Jacobin Magazine on the revolutionary political allegory in "Snowpiercer": 'If Snowpiercer had merely told the tale of an oppressed working class rising up to seize power from an evil overlord, it would already have been an improvement over most of the political messages in mainstream cinema. There are all sorts of nice touches in its portrayal of a declining capitalism that can maintain its ideological legitimacy even when it literally has no more bullets in its guns. But the story Bong tells goes beyond that. It's about the limitations of a revolution which merely takes over the existing social machinery rather than attempting to transcend it. '
Debunking The Dangerous "If You Have Nothing To Hide, You Have Nothing To Fear"
A great resource bookmark from Falkvinge.
There are at least four good reasons to reject this argument solidly and uncompromisingly: The rules may change, it's not you who determine if you're guilty, laws must be broken for society to progress, and privacy is a basic human need.
nsa politics privacy security surveillance gchq rick-falkvinge society
"I have secrets": Ross Ulbricht's private journal shows Silk Road's birth | Ars Technica
Ross Ulbricht is so screwed if this evidence stands up, and it sounds like it will
darknet silk-road ross-ulbricht dread-pirate-roberts fbi bitcoin
Foreign Founders Should Look Beyond Silicon Valley | TechCrunch
'Reasons abound for international entrepreneurs and top technical talent to stay away from Silicon Valley and build their startup somewhere else.' Strongly agreed. This factoid is particularly nuts: 'As Balaji Srinivasan of a16z has observed, roughly 50%+ of the capital allocated for early stage tech investments is actually flowing into Bay Area real estate, directly through office rentals and indirectly via home rentals as a primary driver of skyrocketing salaries.'
salary bay-area silicon-valley usa tech jobs work real-estate rent startups techcrunch
-
A much better carbon-relay, written in C rather than Python. Linking as we've been using it in production for quite a while with no problems.
The main reason to build a replacement is performance and configurability. Carbon is single threaded, and sending metrics to multiple consistent-hash clusters requires chaining of relays. This project provides a multithreaded relay which can address multiple targets and clusters for each and every metric based on pattern matches.
Surveillance of social media not way to fight terrorism – Minister
Blanket surveillance of social media is not the solution to combating terrorism and the rights of the individual to privacy must be protected, Data Protection Minister Dara Murphy said on Monday. [He] said Ireland and the European Union must protect the privacy rights of individuals on social media. "Freedom of expression, freedom of movement, and the protection of privacy are core tenets of the European Union, which must be upheld."
dara-murphy data-protection privacy surveillance europe eu ireland social-media
Amazing comment from a random sysadmin who's been targeted by the NSA
'Here's a story for you. I'm not a party to any of this. I've done nothing wrong, I've never been suspected of doing anything wrong, and I don't know anyone who has done anything wrong. I don't even mean that in the sense of "I pissed off the wrong people but technically haven't been charged." I mean that I am a vanilla, average, 9-5 working man of no interest to anybody. My geographical location is an accident of my birth. Even still, I wasn't accidentally born in a high-conflict area, and my government is not at war. I'm a sysadmin at a legitimate ISP and my job is to keep the internet up and running smoothly. This agency has stalked me in my personal life, undermined my ability to trust my friends attempting to connect with me on LinkedIn, and infected my family's computer. They did this because they wanted to bypass legal channels and spy on a customer who pays for services from my employer. Wait, no, they wanted the ability to potentially spy on future customers. Actually, that is still not accurate - they wanted to spy on everybody in case there was a potentially bad person interacting with a customer. After seeing their complete disregard for anybody else, their immense resources, and their extremely sophisticated exploits and backdoors - knowing they will stop at nothing, and knowing that I was personally targeted - I'll be damned if I can ever trust any electronic device I own ever again. You all rationalize this by telling me that it "isn't surprising", and that I don't live in the [USA,UK] and therefore I have no rights. I just have one question. Are you people even human?'
nsa via:ioerror privacy spying surveillance linkedin sysadmins gchq security
DRI's Unchanged Position on Eircode
'Broadly, they are satisfied with what we are doing' versus: 'We have deep concerns about the Eircode initiative… We want to state clearly that we are not at all 'satisfied' with the postcode that has been designed or the implementation proposals.'
dri ireland eircode postcodes privacy data-protection quotes misrepresentation
-
The young women interns [in one story in this post] worked in a very different way. As I explored their notes, I noticed that ideas were expanded upon, not abandoned. Challenges were identified, but the male language so often heard in Silicon Valley conference rooms - "Well, let me tell you what the problem with that idea is…." - was not in the room. These young women, without men to define the "appropriate business behavior," used different behaviors and came up with a startling and valuable solution. They showed many of the values that exist outside of dominance-based leadership: strategic thinking, intuition, nurturing and relationship building, values-based decision-making and acceptance of other's input. Women need space to be themselves at work. Until people who have created their success by worshipping at the temple of male behavior, like Sheryl Sandberg, learn to value alternate behaviors, the working world will remain a foreign and hostile culture to women. And if we do not continuously work to build corporate cultures where there is room for other behaviors, women will be cast from or abandoned in a world not of our making, where we continuously "just do not fit in," but where we still must go to earn our livings.
sexism misogyny silicon-valley tech work sheryl-sandberg business collaboration
Are you better off running your big-data batch system off your laptop?
Heh, nice trolling.
Here are two helpful guidelines (for largely disjoint populations): If you are going to use a big data system for yourself, see if it is faster than your laptop. If you are going to build a big data system for others, see that it is faster than my laptop. [...] We think everyone should have to do this, because it leads to better systems and better research.
graph coding hadoop spark giraph graph-processing hardware scalability big-data batch algorithms pagerank
BBC uses RIPA terrorism laws to catch TV licence fee dodgers in Northern Ireland
Give them the power, they'll use that power. 'A document obtained under Freedom of Information legislation confirms the BBC's use of RIPA in Northern Ireland. It states: "The BBC may, in certain circumstances, authorise under the Regulation of Investigatory Powers Act 2000 and Regulation of Investigatory Powers (British Broadcasting Corporation) Order 2001 the lawful use of detection equipment to detect unlicensed use of television receivers... the BBC has used detection authorised under this legislation in Northern Ireland."'
ripa privacy bbc tv license-fee uk northern-ireland law scope-creep
Australia tries to ban crypto research – by ACCIDENT • The Register
Researchers are warned off [discussing] 512-bits-plus key lengths, systems "designed or modified to perform cryptanalytic functions, or "designed or modified to use 'quantum cryptography'". [....] "an email to a fellow academic could land you a 10 year prison sentence".
https://twitter.com/_miw/status/556023024009224192 notes 'the DSGL 5A002 defines it as >512bit RSA, >512bit DH, >112 bit ECC and >56 bit symmetric ciphers; weak as fuck i say.'
A Case Study of Toyota Unintended Acceleration and Software Safety
I drive a Toyota, and this is scary stuff. Critical software systems need to be coded with care, and this isn't it -- they don't even have a bug tracking system!
Investigations into potential causes of Unintended Acceleration (UA) for Toyota vehicles have made news several times in the past few years. Some blame has been placed on floor mats and sticky throttle pedals. But, a jury trial verdict was based on expert opinions that defects in Toyota's Electronic Throttle Control System (ETCS) software and safety architecture caused a fatal mishap. This talk will outline key events in the still-ongoing Toyota UA litigation process, and pull together the technical issues that were discovered by NASA and other experts. The results paint a picture that should inform future designers of safety critical software in automobiles and other systems.
toyota safety realtime coding etcs throttle-control nasa code-review embedded
Group warns of postcode project dangers | Irish Examiner
"We have spoken to the National Consumer Agency, logistics companies and Digital Rights Ireland, with which we have had an indepth conversation to see if there is anything in the proposal that might be considered to have an impact on anyone's privacy. Broadly, they are satisfied with what we are doing," [Patricia Cronin, head of the Department of Communications' postcodes division] told the committee. However in his letter, [DRI's] O'Lachtnain said the group "want to state clearly that we are not at all 'satisfied' with the postcode that has been designed or the implementation proposals".
Some nerve!dri nca privacy patricia-cronin goverment postcodes eircode dpc ireland
Of Course 23andMe's Plan Has Been to Sell Your Genetic Data All Along
Today, 23andMe announced what Forbes reports is only the first of ten deals with big biotech companies: Genentech will pay up to $60 million for access to 23andMe's data to study Parkinson's. You think 23andMe was about selling fun DNA spit tests for $99 a pop? Nope, it's been about selling your data all along.
testing ethics dna genentech 23andme parkinsons diseases health privacy
-
Really nice time series dashboarding app. Might consider replacing graphitus with this...
time-series data visualisation graphs ops dashboards facette
Getting good cancer care through 3D printing
This is pretty incredible.
Balzer downloaded a free software program called InVesalius, developed by a research center in Brazil to convert MRI and CT scan data to 3D images. He used it to create a 3D volume rendering from Scott's DICOM images, which allowed him to look at the tumor from any angle. Then he uploaded the files to Sketchfab and shared them with neurosurgeons around the country in the hope of finding one who was willing to try a new type of procedure. Perhaps unsurprisingly, he found the doctor he was looking for at UPMC, where Scott had her thyroid removed. A neurosurgeon there agreed to consider a minimally invasive operation in which he would access the tumor through Scott's left eyelid and remove it using a micro drill. Balzer had adapted the volume renderings for 3D printing and produced a few full-size models of the front section of Scott's skull on his MakerBot. To help the surgeon vet his micro drilling idea and plan the procedure, Balzer packed up one of the models and shipped it off to Pittsburgh.
diy surgery health cancer tumours medicine 3d-printing 3d scanning mri dicom
AWS Tips I Wish I'd Known Before I Started
Some good advice and guidelines (although some are just silly).
Ever liked a film on Facebook? You've given the security services a key to your soul
The researchers started with 86,000 subjects who had filled out the 100-question personality profile – and this, of course, was done as another app on Facebook – and whose personality scores had been matched by algorithms with their Facebook likes. They then found 17,000 who were willing to have a friend or family member take the personality test on their behalf, trying to predict the answers they would give. The results, from most humans, were stunningly inaccurate. Friends, family and co-workers were all less able to predict how someone would fill out a personality test than the algorithms that had been primed with the subject's Facebook likes. With only 10 likes to work on, the computer was more accurate than a work colleague would be. With 150 likes, it described the subject's personality better than a parent or sibling could. And with 300 likes to work on, it was more accurate than a spouse.
likes facebook privacy prism surveillance profiling personality
David Cameron in 'cloud cuckoo land' over encrypted messaging apps ban | Technology | The Guardian
One insider at a major US technology firm told the Guardian that "politicians are fond of asking why it is that tech companies don't base themselves in the UK" ... "I think if you're saying that encryption is the problem, at a time when consumers and businesses see encryption as a very necessary part of trust online, that's a very indicative point of view."
business guardian david-cameron uk-politics crypto ripa messaging internet privacy
Why DNS in OS X 10.10 is broken, and what you can do to fix it | Ars Technica
ffs Apple. (Via Tony Finch)
Schneier on Security: Why Data Mining Won't Stop Terror
A good reference URL to cut-and-paste when "scanning internet traffic for terrorist plots" rears its head:
This unrealistically accurate system will generate 1 billion false alarms for every real terrorist plot it uncovers. Every day of every year, the police will have to investigate 27 million potential plots in order to find the one real terrorist plot per month. Raise that false-positive accuracy to an absurd 99.9999 percent and you're still chasing 2,750 false alarms per day -- but that will inevitably raise your false negatives, and you're going to miss some of those 10 real plots.
Also, Ben Goldacre saying the same thing: http://www.badscience.net/2009/02/datamining-would-be-lovely-if-it-worked/internet scanning filtering specificity statistics data-mining terrorism law nsa gchq false-positives false-negatives
-
The Prime Minister said today that he would stop the use of methods of communication that cannot be read by the security services even if they have a warrant. He said: "In our country, do we want to allow a means of communication between people which […] we cannot read?" He made the connection between encrypted communications tools and letters and phone conversations, both of which can be read by security services in extreme situations and with a warrant from the home secretary.
Is this key escrow for the UK? 30 Fantastic Irish Beers to try in 2015
this is a great shopping list ;)
Personalization at Spotify using Cassandra
Lots and lots of good detail into the Spotify C* setup (via Bill de hOra)
Global Chilling: The Impact of Mass Surveillance on International Writers | PEN American Center
The report's revelations, based on a survey of nearly 800 writers worldwide, are alarming. Concern about surveillance is now nearly as high among writers living in democracies (75%) as among those living in non-democracies (80%). The levels of self-censorship reported by writers living in democratic countries are approaching the levels reported by writers living in authoritarian or semi-democratic countries.
As Islamists Seek To Silence Cartoonists With Guns, Irish Government Also Says Ciúnas
the urgency of repealing the Irish blasphemy legislation cannot now be overstated. The same cartoons that saw their authors murdered for blasphemy recently, would see Irish authors hauled before our courts. The same nations that execute their citizens for blasphemy, wish to promote the wording of the Irish blasphemy legislation through the UN, in order to expand such provisions to more countries. Ireland is the only European country to recently introduce a new blasphemy law. Following the horrific recent events in Paris, let us be the next country to repeal our blasphemy laws.
Registering children: Ireland's Primary Online Database
If you haven't heard about it, it is a compulsory database of the personal information of children, including PPS numbers, ethnicity, race and language skills, to be held for decades and shared across State agencies.
privacy ppsn databases pod ireland children kids primary-schools
-
What if Silicon Valley had emerged from a racially integrated community? Would the technology industry be different? Would we? And what can the technology industry do now to avoid repeating the mistakes of the past?
Amazing article -- this is the best thing I've ever read on TechCrunch: the political history of race in Silicon Valley and East Palo Alto.racism politics history race silicon-valley palo-alto technology us-politics via:burritojustice
Why we don't use a CDN: A story about SPDY and SSL
All of our assets loaded via the CDN [to our client in Australia] in just under 5 seconds. It only took ~2.7s to get those same assets to our friends down under with SPDY. The performance with no CDN blew the CDN performance out of the water. It is just no comparison. In our case, it really seems that the advantages of SPDY greatly outweigh that of a CDN when it comes to speed.
cdn spdy nginx performance web ssl tls optimization multiplexing tcp ops
A World Transfixed by Screens - The Atlantic
Excellent "In Focus" this week -- 'The continued massive growth of connected mobile devices is shaping not only how we communicate with each other, but how we look, behave, and experience the world around us. Smartphones and other handheld devices have become indispensable tools, appendages held at arm's length to record a scene or to snap a selfie. Recent news photos show refugees fleeing war-torn regions holding up their phones as prized possessions to be saved, and relatives of victims lost to a disaster holding up their smartphones to show images of their loved ones to the press. Celebrity selfies, people alone in a crowd with their phones, events obscured by the very devices used to record that event, the brightly lit faces of those bent over their small screens, these are some of the scenes depicted below.'
mobile photography in-focus alan-taylor the-atlantic phones selfies pictures
"Incremental Stream Processing using Computational Conflict-free Replicated Data Types" [paper]
'Unlike existing alternatives, such as stream processing, that favor the execution of arbitrary application code, we want to capture much of the processing logic as a set of known operations over specialized Computational CRDTs, with particular semantics and invariants, such as min/max/average/median registers, accumulators, top-N sets, sorted sets/maps, and so on. Keeping state also allows the system to decrease the amount of propagated information. Preliminary results obtained in a single example show that Titan has an higher throughput when compared with state of the art stream processing systems.'
-
'Turn websites into structured APIs from your browser in seconds' -- next-generation web scraping, recommended by conoro
Following Fire Phone Flop, Big Changes At Amazon's Lab126 | Fast Company | Business + Innovation
as one insider told me, it feels like "Lab126 is in the doghouse" and that "Jeff is taking out his frustration with the failure of the Fire Phone" on upper management.
-
a conceptual model, with accompanying XML schema, that may be used to quantify and exchange complex uncertainties in data. The interoperable model can be used to describe uncertainty in a variety of ways including: Samples Statistics including mean, variance, standard deviation and quantile Probability distributions including marginal and joint distributions and mixture models
-
How to secure SSH, disabling insecure ciphers etc. (via Padraig)
-
Make "Paste and Match Style" the default, as it should be
Introducing practical and robust anomaly detection in a time series
Twitter open-sources an anomaly-spotting R package:
Early detection of anomalies plays a key role in ensuring high-fidelity data is available to our own product teams and those of our data partners. This package helps us monitor spikes in user engagement on the platform surrounding holidays, major sporting events or during breaking news. Beyond surges in social engagement, exogenic factors – such as bots or spammers – may cause an anomaly in number of favorites or followers. The package can be used to find such bots or spam, as well as detect anomalies in system metrics after a new software release. We're open-sourcing AnomalyDetection because we'd like the public community to evolve the package and learn from it as we have.
statistics twitter r anomaly-detection outliers metrics time-series spikes holt-winters
Mantis: Netflix's Event Stream Processing System
Rx/reactive in style, autoscaling, support for queue/broker-based strong consistency as well as TCP-based lossy delivery
-
'I now a man with a wooden leg named sea what was the name of the other leg SAND'
-
Fergal Crehan's new gig -- good idea!
The Hit Team helps you fight back against leaked photos and videos, internet targeting and revenge porn.
revenge-porn revenge law privacy porn leaks photos videos images selfies
F1: A Distributed SQL Database That Scales
Beyond the interesting-enough stuff about scalability in a distributed SQL store, there's this really nifty point about avoiding the horrors of the SQL/ORM impedance mismatch:
At Google, Protocol Buffers are ubiquitous for data storage and interchange between applications. When we still had a MySQL schema, users often had to write tedious and error-prone transformations between database rows and in-memory data structures. Putting protocol buffers in the schema removes this impedance mismatch and gives users a universal data structure they can use both in the database and in application code…. Protocol Buffer columns are more natural and reduce semantic complexity for users, who can now read and write their logical business objects as atomic units, without having to think about materializing them using joins across several tables.
This is something that pretty much any store can already adopt. Go protobufs. (or Avro, etc.) Also, I find this really neat, and I hope this idea is implemented elsewhere soon: asynchronous schema updates:Schema changes are applied asynchronously on multiple F1 servers. Anomalies are prevented by the use of a schema leasing mechanism with support for only current and next schema versions; and by subdividing schema changes into multiple phases where consecutive pairs of changes are mutually compatible and cannot cause anomalies.
Avleen Vig on distributed engineering teams
This is a really excellent post on the topic, rebutting Paul Graham's Bay-Area-centric thoughts on the topic very effectively. I've worked in both distributed and non-distributed, as well as effective and ineffective teams ;), and Avleen's thoughts are very much on target.
I've been involved in the New York start up scene since I joined Etsy in 2010. Since that time, I've seen more and more companies there embrace having distributed teams. Two companies I know which have risen to the top while doing this have been Etsy and DigitalOcean. Both have exceptional engineering teams working on high profile products used by many, many people around the world. There are certainly others outside New York, including Automattic, GitHub, Chef Inc, Puppet... the list goes on. So how did this happen? And why do people continue to insist that distributed teams lower performance, and are a bad idea? Partly because we've done a poor job of showing our industry how to be successful at it, and partly because it's hard. Having successful distributed teams requires special skills from management, which arent't easily learned until you have to manage a distributed team. Catch 22.
business culture management communication work distributed-teams avleen-vig engineering
Hack workaround to get JVM thread priorities working on Linux
As used in Cassandra ( http://grokbase.com/t/hbase/dev/13bf9kezes/about-xx-threadprioritypolicy-42 )!
if you just set the "ThreadPriorityPolicy" to something else than the legal values 0 or 1, [...] a slight logic bug in Sun's JVM code kicks in, and thus sets the policy to be as if running with root - thus you get exactly what one desire. The operating system, Linux, won't allow priorities to be heightened above "Normal" (negative nice value), and thus just ignores those requests (setting it to normal instead, nice value 0) - but it lets through the requests to set it lower (setting the nice value to some positive value).
cassandra thread-priorities threads java jvm linux nice hacks
Amiko Alien2 / Enigma Discussion Thread - boards.ie
Enigma is a Linux based alternative to the default Spark operating system on these boxes. Enigma is a more customisable OS and provides the ability to add plugins which can accomplish many tasks enabling users to have a box which might look and perform like a Sky box, giving a 7 day EPG and an alternative to series link.
Looks like a pretty solid hacker community...Hague reassures MPs on Office 365 data storage as Microsoft ordered to hand over email data
William Hague, the leader of the House of Commons, has responded to concerns raised by an MP about the security of parliamentary data stored on Microsoft's Cloud-based servers in Europe. "The relevant servers are situated in the Republic of Ireland and the Netherlands, both being territories covered by the EC Data Protection Directive," William Hague wrote in a letter to John Hemming, MP for Birmingham Yardley. "Any access by US authorities to such data would have to be by way of mutual legal assistance arrangements with those countries." [...] John Hemming MP told Computer Weekly Hague's reassurances carried little weight in the face of aggressive legal action by the US government. "The Microsoft case makes it clear that, in the end, the fact that Microsoft is a US company legally trumps the European Data Protection Directive [...] and where [the letter says] the US authorities could not exercise a right of search and seizure on an extraterritorial basis, well, they are doing that, in America, today."
Sounds like they didn't think that through...mail privacy parliament office-365 microsoft mlat surveillance
Why Sweden Has the World's Safest Roads
Nearly half the EU-wide average.
Sweden has also created 12,600 safer pedestrian crossings with features such as bridges, flashing lights, and speed bumps. That's estimated to have halved pedestrian deaths over the past five years. The country has lowered speed limits in urban, crowded areas and built barriers to protect bikers from incoming traffic. A crackdown on drunk driving has also helped.
sweden safety engineering road-safety pedestrian roads cycling
Do not use 'YYYY' or '%G' in time format specifiers
Formats the year based on ISO week numbering, which often is not what you want. Both have been responsible for high-profile production bugs (in Apple and Android).
apple android bugs time date year iso week formatting strftime posix
Indian Government blocks 32 Sites, including GitHub, Pastebin, Imgur and Vimeo
Spectacularly inept. Pretty much every UGC site there is
ugc india filtering blocking terrorism isis github vimeo pastebin censorship
Why Sweden Has the World's Safest Roads
Nearly half the EU-wide average.
Sweden has also created 12,600 safer pedestrian crossings with features such as bridges, flashing lights, and speed bumps. That's estimated to have halved pedestrian deaths over the past five years. The country has lowered speed limits in urban, crowded areas and built barriers to protect bikers from incoming traffic. A crackdown on drunk driving has also helped.
sweden safety engineering road-safety pedestrian roads cycling
Do not use 'YYYY' or '%G' in time format specifiers
Formats the year based on ISO week numbering, which often is not what you want. Both have been responsible for high-profile production bugs (in Apple and Android).
apple android bugs time date year iso week formatting strftime posix
Indian Government blocks 32 Sites, including GitHub, Pastebin, Imgur and Vimeo
Spectacularly inept. Pretty much every UGC site there is
ugc india filtering blocking terrorism isis github vimeo pastebin censorship
The open-office trend is destroying the workplace
Wow, where has this person been for the past 20 years that they haven't had to encounter this? I can only imagine having a private office, tbh.
my personal performance at work has hit an all-time low. Each day, my associates and I are seated at a table staring at each other, having an ongoing 12-person conversation from 9 a.m. to 5 p.m. It's like being in middle school with a bunch of adults. Those who have worked in private offices for decades have proven to be the most vociferous and rowdy. They haven't had to consider how their loud habits affect others, so they shout ideas at each other across the table and rehash jokes of yore. As a result, I can only work effectively during times when no one else is around, or if I isolate myself in one of the small, constantly sought-after, glass-windowed meeting rooms around the perimeter.
The open-office trend is destroying the workplace
Wow, where has this person been for the past 20 years that they haven't had to encounter this? I can only imagine having a private office, tbh.
my personal performance at work has hit an all-time low. Each day, my associates and I are seated at a table staring at each other, having an ongoing 12-person conversation from 9 a.m. to 5 p.m. It's like being in middle school with a bunch of adults. Those who have worked in private offices for decades have proven to be the most vociferous and rowdy. They haven't had to consider how their loud habits affect others, so they shout ideas at each other across the table and rehash jokes of yore. As a result, I can only work effectively during times when no one else is around, or if I isolate myself in one of the small, constantly sought-after, glass-windowed meeting rooms around the perimeter.