Changelog¶
2.23.0 / 2020-08-14¶
Array¶
Reduce
np.zeros
,ones
, andfull
array size with broadcasting (GH#6491) Matthias BussonnierAdd missing
meta=
fortrim
inmap_overlap
(GH#6494) Peter Andreas Entschev
Bag¶
Bag repartition partition size (GH#6371) joshreback
Core¶
Scalar.__dask_layers__()
to returnself._name
instead ofself.key
(GH#6507) Mads R. B. KristensenUpdate dependencies correctly in
fuse_root
optimization (GH#6508) Mads R. B. Kristensen
DataFrame¶
Adds
items
to dataframe (GH#6503) Thomas J FanInclude compression in
write_table
call (GH#6499) Julia SignellFixed warning in
nonempty_series
(GH#6485) Tom AugspurgerIntelligently determine partitions based on type of first arg (GH#6479) Matthew Rocklin
Fix pyarrow
mkdirs
(GH#6475) Julia SignellFix duplicate parquet output in
to_parquet
(GH#6451) michaelnarodovitch
Documentation¶
Fix documentation
da.histogram
(GH#6439) Roberto PanaiFixed a few typos in the SQL docs (GH#6489) Mike McCarty
Docs for SQLing (GH#6453) Martin Durant
2.22.0 / 2020-07-31¶
Array¶
Compatibility for NumPy dtype deprecation (GH#6430) Tom Augspurger
Core¶
Implement
sizeof
for somebytes
-like objects (GH#6457) John A KirkhamHTTP error for new
fsspec
(GH#6446) Martin DurantWhen
RecursionError
is raised, return uuid fromtokenize
function (GH#6437) Julia SignellInstall deps of upstream-dev packages (GH#6431) Tom Augspurger
DataFrame¶
Add single quotes around column names if strings (GH#6471) Gil Forsyth
Refactor
ArrowEngine
for betterread_parquet
performance (GH#6346) Richard (Rick) ZamoraAdd
tolist
dispatch (GH#6444) GALI PREM SAGARCompatibility with pandas 1.1.0rc0 (GH#6429) Tom Augspurger
Multi value pivot table (GH#6428) joshreback
Duplicate argument definitions in
to_csv
docstring (GH#6411) Jun Han (Johnson) Ooi
Documentation¶
Add utility to docs to convert YAML config to env vars and back (GH#6472) Jacob Tomlinson
Fix parameter server rendering (GH#6466) Scott Sievert
Fixes broken links (GH#6403) Jim Circadian
Complete parameter server implementation in docs (GH#6449) Scott Sievert
Fix typo (GH#6436) Jack Xiaosong Xu
2.21.0 / 2020-07-17¶
Array¶
Correct error message in
array.routines.gradient()
(GH#6417) johnomotaniFix blockwise concatenate for array with some
dimension=1
(GH#6342) Matthias Bussonnier
Bag¶
Fix
bag.take
example (GH#6418) Roberto Panai
Core¶
Groups values in optimization pass should only be graph and keys – not an optimization + keys (GH#6409) Ben Zaitlen
Call custom optimizations once, with
kwargs
provided (GH#6382) Clark ZinzowInclude
pickle5
for testing on Python 3.7 (GH#6379) John A Kirkham
DataFrame¶
Correct typo in error message (GH#6422) Tom McTiernan
Use
pytest.warns
to check forUserWarning
(GH#6378) Richard (Rick) ZamoraParse
bytes_per_chunk keyword
from string (GH#6370) Matthew Rocklin
Documentation¶
Numpydoc formatting (GH#6421) Matthias Bussonnier
Unpin
numpydoc
following 1.1 release (GH#6407) Gil ForsythNumpydoc formatting (GH#6402) Matthias Bussonnier
Add instructions for using conda when installing code for development (GH#6399) Ray Bell
2.20.0 / 2020-07-02¶
Array¶
Register
sizeof
for numpy zero-strided arrays (GH#6343) Matthias BussonnierUse
concatenate_lookup
inconcatenate
(GH#6339) John A KirkhamFix rechunking of arrays with some zero-length dimensions (GH#6335) Matthias Bussonnier
DataFrame¶
Dispatch
iloc`
calls togetitem
(GH#6355) Gil ForsythHandle unnamed pandas
RangeIndex
in fastparquet engine (GH#6350) Richard (Rick) ZamoraPreserve index when writing partitioned parquet datasets with pyarrow (GH#6282) Richard (Rick) Zamora
Use
ignore_index
for pandas’group_split_dispatch
(GH#6251) Richard (Rick) Zamora
2.19.0 / 2020-06-19¶
Array¶
Cast chunk sizes to python int
dtype
(GH#6326) Gil ForsythAdd
shape=None
to*_like()
array creation functions (GH#6064) Anderson Banihirwe
Core¶
Update expected error msg for protocol difference in fsspec (GH#6331) Gil Forsyth
Fix for floats < 1 in
parse_bytes
(GH#6311) Gil ForsythFix exception causes all over the codebase (GH#6308) Ram Rachum
Fix duplicated tests (GH#6303) James Lamb
Remove unused testing function (GH#6304) James Lamb
DataFrame¶
Add high-level CSV Subgraph (GH#6262) Gil Forsyth
Fix
ValueError
when merging an index-only 1-partition dataframe (GH#6309) Krishan BhasinMake
index.map
clear divisions. (GH#6285) Julia Signell
Documentation¶
Add link to 2020 survey (GH#6328) Tom Augspurger
Update
bag.rst
(GH#6317) Ben Shaver
2.18.1 / 2020-06-09¶
Array¶
Don’t try to set name on
full
(GH#6299) Julia SignellHistogram: support lazy values for range/bins (another way) (GH#6252) Gabe Joseph
Core¶
Fix exception causes in
utils.py
(GH#6302) Ram RachumImprove performance of
HighLevelGraph
construction (GH#6293) Julia Signell
Documentation¶
Now readthedocs builds unrelased features’ docstrings (GH#6295) Antonio Ercole De Luca
Add
asyncssh
intersphinx mappings (GH#6298) Jacob Tomlinson
2.18.0 / 2020-06-05¶
Array¶
Cast slicing index to dask array if same shape as original (GH#6273) Julia Signell
Fix
stack
error message (GH#6268) Stephanie Gottfull
&full_like
: error on non-scalarfill_value
(GH#6129) HuiteSupport for multiple arrays in
map_overlap
(GH#6165) Eric CzechPad resample divisions so that edges are counted (GH#6255) Julia Signell
Bag¶
Random sampling of k elements from a dask bag #4799 (GH#6239) Antonio Ercole De Luca
DataFrame¶
Add
dropna
,sort
, andascending
tosort_values
(GH#5880) Julia SignellGeneralize
from_dask_array
(GH#6263) GALI PREM SAGARAdd derived docstring for
SeriesGroupby.nunique
(GH#6284) Julia SignellRemove
NotImplementedError
in resample with rule (GH#6274) Abdulelah Bin MahfoodhAdd
dd.to_sql
(GH#6038) Ryan Williams
2.17.2 / 2020-05-28¶
Core¶
Re-add the
complete
extra (GH#6257) Jim Crist-Harif
DataFrame¶
Raise error if
resample
isn’t going to give right answer (GH#6244) Julia Signell
2.17.1 / 2020-05-28¶
Array¶
Empty array rechunk (GH#6233) Andrew Fulton
Core¶
Make
pyyaml
required (GH#6250) Jim Crist-HarifFix install commands from
ImportError
(GH#6238) Gaurav SheniRemove issue template (GH#6249) Jacob Tomlinson
DataFrame¶
Pass
ignore_index
todd_shuffle
fromDataFrame.shuffle
(GH#6247) Richard (Rick) ZamoraCope with missing HDF keys (GH#6204) Martin Durant
Generalize
describe
&quantile
apis (GH#5137) GALI PREM SAGAR
2.17.0 / 2020-05-26¶
Array¶
Bag¶
Random Choice on Bags (GH#6208) Antonio Ercole De Luca
Core¶
Raise warning
delayed.visualise()
(GH#6216) Amol UmbarkarEnsure other pickle arguments work (GH#6229) John A Kirkham
Overhaul
fuse()
config (GH#6198) Guido ImperialeUpdate
dask.order.order
to consider “next” nodes using both FIFO and LIFO (GH#5872) Erik Welch
DataFrame¶
Use 0 as
fill_value
for more agg methods (GH#6245) Julia SignellGeneralize
rearrange_by_column_tasks
and addDataFrame.shuffle
(GH#6066) Richard (Rick) ZamoraXfail
test_rolling_numba_engine
for newer numba and older pandas (GH#6236) James BourbeauGeneralize
fix_overlap
(GH#6240) GALI PREM SAGARAvoid shuffle when setting a presorted index with overlapping divisions (GH#6226) Krishan Bhasin
Adjust the Parquet engine classes to allow more easily subclassing (GH#6211) Marius van Niekerk
Fix
dd.merge_asof
withleft_on='col'
&right_index=True
(GH#6192) noreentryMove
AUTO_BLOCKSIZE
out ofread_csv
signature (GH#6214) Jim Crist-Harif.loc
indexing with callable (GH#6185) Endre Mark BorzaAvoid apply in
_compute_sum_of_squares
for groupby std agg (GH#6186) Richard (Rick) ZamoraMinor correction to
test_parquet
(GH#6190) Brian LarsenAdhering to the passed pat for delimeter join and fix error message (GH#6194) GALI PREM SAGAR
Skip
test_to_parquet_with_get
if no parquet libs available (GH#6188) Scott Sanderson
Documentation¶
Added documentation for
distributed.Event
class (GH#6231) Nils Braun
2.16.0 / 2020-05-08¶
Array¶
Fix array general-reduction name (GH#6176) Nick Evans
Replace
dim
withshape
inunravel_index
(GH#6155) Julia SignellMoment: handle all elements being masked (GH#5339) Gabe Joseph
Core¶
Remove Redundant string concatenations in dask code-base (GH#6137) GALI PREM SAGAR
Upstream compat (GH#6159) Tom Augspurger
Ensure
sizeof
of dict and sequences returns an integer (GH#6179) James BourbeauEstimate python collection sizes with random sampling (GH#6154) Florian Jetter
Update test upstream (GH#6146) Tom Augspurger
Skip test for mindeps build (GH#6144) Tom Augspurger
Switch default multiprocessing context to “spawn” (GH#4003) Itamar Turner-Trauring
Update manifest to include dask-schema (GH#6140) Ben Zaitlen
DataFrame¶
Harden inconsistent-schema handling in pyarrow-based
read_parquet
(GH#6160) Richard (Rick) ZamoraAdd compute
kwargs
to methods that write data to disk (GH#6056) Krishan BhasinFix issue where
unique
returns an index like result from backends (GH#6153) GALI PREM SAGARFix internal error in
map_partitions
with collections (GH#6103) Tom Augspurger
Documentation¶
Add phase of computation to index TOC (GH#6157) Ben Zaitlen
Remove unused imports in scheduling script (GH#6138) James Lamb
Fix indent (GH#6147) Martin Durant
Add Tom’s log config example (GH#6143) Martin Durant
2.15.0 / 2020-04-24¶
Array¶
Update
dask.array.from_array
to warn when passed a Dask collection (GH#6122) James BourbeauUn-numpy like behaviour in
dask.array.pad
(GH#6042) Mark BoerAdd support for
repeats=0
inda.repeat
(GH#6080) James Bourbeau
Core¶
Fix yaml layout for schema (GH#6132) Ben Zaitlen
Configuration Reference (GH#6069) Ben Zaitlen
Add configuration option to turn off task fusion (GH#6087) Matthew Rocklin
Skip pyarrow on windows (GH#6094) Tom Augspurger
Set limit to maximum length of fused key (GH#6057) Lucas Rademaker
Add test against #6062 (GH#6072) Martin Durant
Bump checkout action to v2 (GH#6065) James Bourbeau
DataFrame¶
Generalize categorical calls to support cudf
Categorical
(GH#6113) GALI PREM SAGARAvoid reading
_metadata
on every worker (GH#6017) Richard (Rick) ZamoraUse
group_split_dispatch
andignore_index
inapply_concat_apply
(GH#6119) Richard (Rick) ZamoraHandle new (dtype) pandas metadata with pyarrow (GH#6090) Richard (Rick) Zamora
Skip
test_partition_on_cats_pyarrow
if pyarrow is not installed (GH#6112) James BourbeauUpdate DataFrame len to handle columns with the same name (GH#6111) James Bourbeau
ArrowEngine
bug fixes and test coverage (GH#6047) Richard (Rick) ZamoraAdded mode (GH#5958) Adam Lewis
Documentation¶
Extend preload documentation (GH#6077) Matthew Rocklin
Fixed small typo in DataFrame
map_partitions()
docstring (GH#6115) Eugene HuangFix typo: “double” should be times, not plus (GH#6091) David Chudzicki
Fix first line of
array.random.*
docs (GH#6063) Martin DurantAdd section about
Semaphore
in distributed (GH#6053) Florian Jetter
2.14.0 / 2020-04-03¶
Array¶
Added
np.iscomplexobj
implementation (GH#6045) Tom Augspurger
Core¶
Update
test_rearrange_disk_cleanup_with_exception
to pass without cloudpickle installed (GH#6052) James BourbeauFixed flaky
test-rearrange
(GH#5977) Tom Augspurger
DataFrame¶
Use
_meta_nonempty
for dtype casting instack_partitions
(GH#6061) mlondschienFix bugs in
_metadata
creation and filtering in parquetArrowEngine
(GH#6023) Richard (Rick) Zamora
Documentation¶
DOC: Add name caveats (GH#6040) Tom Augspurger
2.13.0 / 2020-03-25¶
Array¶
Support
dtype
and other keyword arguments inda.random
(GH#6030) Matthew RocklinRegister support for
cupy
sparsehstack
/vstack
(GH#5735) Corey J. NoletForce
self.name
tostr
indask.array
(GH#6002) Chuanzhu Xu
Bag¶
Set
rename_fused_keys
toNone
by default inbag.optimize
(GH#6000) Lucas Rademaker
Core¶
Copy dict in
to_graphviz
to prevent overwriting (GH#5996) JulianWgsStricter pandas
xfail
(GH#6024) Tom AugspurgerFix CI failures (GH#6013) James Bourbeau
Update
toolz
to 0.8.2 and usetlz
(GH#5997) Ryan GroutMove Windows CI builds to GitHub Actions (GH#5862) James Bourbeau
DataFrame¶
Improve path-related exceptions in
read_hdf
(GH#6032) psimajFix
dtype
handling indd.concat
(GH#6006) mlondschienHandle cudf’s leftsemi and leftanti joins (GH#6025) Richard J Zamora
Remove unused
npartitions
variable indd.from_pandas
(GH#6019) Daniel Saxton
Documentation¶
Fix indentation in scheduler-overview docs (GH#6022) Matthew Rocklin
Update task graphs in optimize docs (GH#5928) Julia Signell
Optionally get rid of intermediary boxes in visualize, and add more labels (GH#5976) Julia Signell
2.12.0 / 2020-03-06¶
Array¶
Improve reuse of temporaries with numpy (GH#5933) Bruce Merry
Make
map_blocks
withblock_info
produce aBlockwise
(GH#5896) Bruce MerryOptimize
make_blockwise_graph
(GH#5940) Bruce MerryFix axes ordering in
da.tensordot
(GH#5975) Gil ForsythAdds empty mode to
array.pad
(GH#5931) Thomas J Fan
Core¶
Remove
toolz.memoize
dependency indask.utils
(GH#5978) Ryan GroutClose pool leaking subprocess (GH#5979) Tom Augspurger
Pin
numpydoc
to0.8.0
(fix double autoescape) (GH#5961) Gil ForsythRegister deterministic tokenization for
range
objects (GH#5947) James BourbeauUnpin
msgpack
in CI (GH#5930) JAmes BourbeauEnsure dot results are placed in unique files. (GH#5937) Elliott Sales de Andrade
Add remaining optional dependencies to Travis 3.8 CI build environment (GH#5920) James Bourbeau
DataFrame¶
Skip parquet
getitem
optimization for some keys (GH#5917) Tom AugspurgerAdd
ignore_index
argument torearrange_by_column
code path (GH#5973) Richard J ZamoraAdd DataFrame and Series
memory_usage_per_partition
methods (GH#5971) James Bourbeauxfail
test_describe when using Pandas 0.24.2 (GH#5948) James BourbeauImplement
dask.dataframe.to_numeric
(GH#5929) Julia SignellAdd new error message content when columns are in a different order (GH#5927) Julia Signell
Use shallow copy for assign operations when possible (GH#5740) Richard J Zamora
Documentation¶
Changed above to below in
dask.array.triu
docs (GH#5984) Henrik AnderssonArray slicing: fix typo in
slice_with_int_dask_array
error message (GH#5981) Gabe JosephGrammar and formatting updates to docstrings (GH#5963) James Lamb
Update title of DataFrame extension docs (GH#5954) James Bourbeau
Fixed typos in documentation (GH#5962) James Lamb
Add original class or module as a
kwarg
on_bind_*
methods (GH#5946) Julia SignellUpdate optimization doc for python 3 (GH#5926) Julia Signell
2.11.0 / 2020-02-19¶
Array¶
Cache result of
Array.shape
(GH#5916) Bruce MerryImprove accuracy of
estimate_graph_size
forrechunk
(GH#5907) Bruce MerrySkip rechunk steps that do not alter chunking (GH#5909) Bruce Merry
Support
dtype
and otherkwargs
incoarsen
(GH#5903) Matthew RocklinPush chunk override from
map_blocks
into blockwise (GH#5895) Bruce MerryAvoid using
rewrite_blockwise
for a singleton (GH#5890) Bruce MerryOptimize
slices_from_chunks
(GH#5891) Bruce MerryAvoid unnecessary
__getitem__
inblock()
when chunks have correct dimensionality (GH#5884) Thomas Robitaille
Bag¶
Add
include_path
option fordask.bag.read_text
(GH#5836) Yifan GuFixes
ValueError
in delayed execution of bagged NumPy array (GH#5828) Surya Avala
Core¶
CI: Pin
msgpack
(GH#5923) Tom AugspurgerRename
test_inner
totest_outer
(GH#5922) Shiva Raisinghaniquote
should quote dicts too (GH#5905) Bruce MerryRegister a normalizer for literal (GH#5898) Bruce Merry
Improve layer name synthesis for non-HLGs (GH#5888) Bruce Merry
Replace flake8 pre-commit-hook with upstream (GH#5892) Julia Signell
Call pip as a module to avoid warnings (GH#5861) Cyril Shcherbin
Close
ThreadPool
at exit (GH#5852) Tom AugspurgerRemove
dask.dataframe
import in tokenization code (GH#5855) James Bourbeau
DataFrame¶
Require
pandas>=0.23
(GH#5883) Tom AugspurgerRemove lambda from dataframe aggregation (GH#5901) Matthew Rocklin
Fix exception chaining in
dataframe/__init__.py
(GH#5882) Ram RachumAdd support for reductions on empty dataframes (GH#5804) Shiva Raisinghani
Expose
sort=
argument for groupby (GH#5801) Richard J ZamoraUse parquet read speed-ups from
fastparquet.api.paths_to_cats
. (GH#5821) Igor Gotlibovych
Documentation¶
Deprecate
doc_wraps
(GH#5912) Tom AugspurgerUpdate array internal design docs for HighLevelGraph era (GH#5889) Bruce Merry
Move over dashboard connection docs (GH#5877) Matthew Rocklin
Move prometheus docs from distributed.dask.org (GH#5876) Matthew Rocklin
Removing duplicated DO block at the end (GH#5878) K.-Michael Aye
map_blocks
see also (GH#5874) Tom AugspurgerMore derived from (GH#5871) Julia Signell
Fix typo (GH#5866) Yetunde Dada
Fix typo in
cloud.rst
(GH#5860) Andrew ThomasAdd note pointing to code of conduct and diversity statement (GH#5844) Matthew Rocklin
2.10.1 / 2020-01-30¶
Fix Pandas 1.0 version comparison (GH#5851) Tom Augspurger
Fix typo in distributed diagnostics documentation (GH#5841) Gerrit Holl
2.10.0 / 2020-01-28¶
Support for pandas 1.0’s new
BooleanDtype
andStringDtype
(GH#5815) Tom AugspurgerCompatibility with pandas 1.0’s API breaking changes and deprecations (GH#5792) Tom Augspurger
Fixed non-deterministic tokenization of some extension-array backed pandas objects (GH#5813) Tom Augspurger
Fixed handling of dataclass class objects in collections (GH#5812) Matteo De Wint
Fixed resampling with tz-aware dates when one of the endpoints fell in a non-existent time (GH#5807) dfonnegra
Delay initial Zarr dataset creation until the computation occurs (GH#5797) Chris Roat
Use parquet dataset statistics in more cases with the
pyarrow
engine (GH#5799) Richard J ZamoraFixed exception in
groupby.std()
when some of the keys were large integers (GH#5737) H. Thomson Comer
2.9.2 / 2020-01-16¶
Array¶
Unify chunks in
broadcast_arrays
(GH#5765) Matthew Rocklin
Core¶
xfail
CSV encoding tests (GH#5791) Tom AugspurgerUpdate order to handle empty dask graph (GH#5789) James Bourbeau
Redo
dask.order.order
(GH#5646) Erik Welch
DataFrame¶
Add transparent compression for on-disk shuffle with
partd
(GH#5786) Christian WespFix
repr
for empty dataframes (GH#5781) Shiva RaisinghaniPandas 1.0.0RC0 compat (GH#5784) Tom Augspurger
Remove buggy assertions (GH#5783) Tom Augspurger
Pandas 1.0 compat (GH#5782) Tom Augspurger
Fix bug in pyarrow-based
read_parquet
on partitioned datasets (GH#5777) Richard J ZamoraCompat for pandas 1.0 (GH#5779) Tom Augspurger
Fix groupby/mean error with with categorical index (GH#5776) Richard J Zamora
Support empty partitions when performing cumulative aggregation (GH#5730) Matthew Rocklin
set_index
accepts single-item unnested list (GH#5760) Wes RoachFixed partitioning in set index for ordered
Categorical
(GH#5715) Tom Augspurger
Documentation¶
Note additional use case for
normalize_token.register
(GH#5766) Thomas A CaswellSmall typos (GH#5771) Maarten Breddels
Fix typo in Task Expectations docs (GH#5767) James Bourbeau
Add docs section on task expectations to graph page (GH#5764) Devin Petersohn
2.9.1 / 2019-12-27¶
Array¶
Support Array.view with dtype=None (GH#5736) Anderson Banihirwe
Add dask.array.nanmedian (GH#5684) Deepak Cherian
Core¶
xfail test_temporary_directory on Python 3.8 (GH#5734) James Bourbeau
Add support for Python 3.8 (GH#5603) James Bourbeau
Use id to dedupe constants in rewrite_blockwise (GH#5696) Jim Crist
DataFrame¶
Raise error when converting a dask dataframe scalar to a boolean (GH#5743) James Bourbeau
Ensure dataframe groupby-variance is greater than zero (GH#5728) Matthew Rocklin
Fix DataFrame.__iter__ (GH#5719) Tom Augspurger
Support Parquet filters in disjunctive normal form, like PyArrow (GH#5656) Matteo De Wint
Auto-detect categorical columns in ArrowEngine-based read_parquet (GH#5690) Richard J Zamora
Skip parquet getitem optimization tests if no engine found (GH#5697) James Bourbeau
Fix independent optimization of parquet-getitem (GH#5613) Tom Augspurger
Documentation¶
Link to examples.dask.org in several places (GH#5733) Tom Augspurger
Add missing ” in performance report example (GH#5724) James Bourbeau
Resolve several documentation build warnings (GH#5685) James Bourbeau
add info on performance_report (GH#5713) Ben Zaitlen
Add more docs disclaimers (GH#5710) Julia Signell
Update numpydoc dependency (GH#5694) James Bourbeau
2.9.0 / 2019-12-06¶
Array¶
Fix
da.std
to work with NumPy arrays (GH#5681) James Bourbeau
Core¶
Register
sizeof
functions for Numba and RMM (GH#5668) John A KirkhamUpdate meeting time (GH#5682) Tom Augspurger
DataFrame¶
Modify
dd.DataFrame.drop
to use shallow copy (GH#5675) Richard J ZamoraFix bug in
_get_md_row_groups
(GH#5673) Richard J ZamoraClose sqlalchemy engine after querying DB (GH#5629) Krishan Bhasin
Allow
dd.map_partitions
to not enforce meta (GH#5660) Matthew RocklinGeneralize
concat_unindexed_dataframes
to support cudf-backend (GH#5659) Richard J ZamoraAdd dataframe resample methods (GH#5636) Ben Zaitlen
Compute length of dataframe as length of first column (GH#5635) Matthew Rocklin
Documentation¶
Doc fixup (GH#5665) James Bourbeau
Update doc build instructions (GH#5640) James Bourbeau
Add documentation build (GH#5617) James Bourbeau
2.8.1 / 2019-11-22¶
Array¶
Use auto rechunking in
da.rechunk
if no value given (GH#5605) Matthew Rocklin
Core¶
Add simple action to activate GH actions (GH#5619) James Bourbeau
DataFrame¶
Fix “file_path_0” bug in
aggregate_row_groups
(GH#5627) Richard J ZamoraAdd
chunksize
argument toread_parquet
(GH#5607) Richard J ZamoraChange
test_repartition_npartitions
to support arch64 architecture (GH#5620) ossdev07Categories lost after groupby + agg (GH#5423) Oliver Hofkens
Fixed relative path issue with parquet metadata file (GH#5608) Nuno Gomes Silva
Enable gpu-backed covariance/correlation in dataframes (GH#5597) Richard J Zamora
Documentation¶
Fix institutional faq and unknown doc warnings (GH#5616) James Bourbeau
Add doc for some utils (GH#5609) Tom Augspurger
Removes
html_extra_path
(GH#5614) James BourbeauFixed See Also referencence (GH#5612) Tom Augspurger
2.8.0 / 2019-11-14¶
Array¶
Implement complete dask.array.tile function (GH#5574) Bouwe Andela
Add median along an axis with automatic rechunking (GH#5575) Matthew Rocklin
Allow da.asarray to chunk inputs (GH#5586) Matthew Rocklin
Bag¶
Use key_split in Bag name (GH#5571) Matthew Rocklin
Core¶
Switch Doctests to Py3.7 (GH#5573) Ryan Nazareth
Relax get_colors test to adapt to new Bokeh release (GH#5576) Matthew Rocklin
Add dask.blockwise.fuse_roots optimization (GH#5451) Matthew Rocklin
Add sizeof implementation for small dicts (GH#5578) Matthew Rocklin
Update fsspec, gcsfs, s3fs (GH#5588) Tom Augspurger
DataFrame¶
Add dropna argument to groupby (GH#5579) Richard J Zamora
Revert “Remove import of dask_cudf, which is now a part of cudf (GH#5568)” (GH#5590) Matthew Rocklin
Documentation¶
Add best practice for dask.compute function (GH#5583) Matthew Rocklin
Create FUNDING.yml (GH#5587) Gina Helfrich
Add screencast for coordination primitives (GH#5593) Matthew Rocklin
Move funding to .github repo (GH#5589) Tom Augspurger
Update calendar link (GH#5569) Tom Augspurger
2.7.0 / 2019-11-08¶
This release drops support for Python 3.5
Array¶
Update da.array to always return a dask array (GH#5510) James Bourbeau
Skip transpose on trivial inputs (GH#5523) Ryan Abernathey
Avoid NumPy scalar string representation in tokenize (GH#5527) James Bourbeau
Remove unnecessary tiledb shape constraint (GH#5545) Norman Barker
Removes bytes from sparse array HTML repr (GH#5556) James Bourbeau
Core¶
Drop Python 3.5 (GH#5528) James Bourbeau
Update the use of fixtures in distributed tests (GH#5497) Matthew Rocklin
Changed deprecated bokeh-port to dashboard-address (GH#5507) darindf
Avoid updating with identical dicts in ensure_dict (GH#5501) James Bourbeau
Test Upstream (GH#5516) Tom Augspurger
Accelerate reverse_dict (GH#5479) Ryan Grout
Update test_imports.sh (GH#5534) James Bourbeau
Support cgroups limits on cpu count in multiprocess and threaded schedulers (GH#5499) Albert DeFusco
Update minimum pyarrow version on CI (GH#5562) James Bourbeau
Make cloudpickle optional (GH#5511) Guido Imperiale
DataFrame¶
Add an example of index_col usage (GH#3072) Bruno Bonfils
Explicitly use iloc for row indexing (GH#5500) Krishan Bhasin
Accept dask arrays on columns assignemnt (GH#5224) Henrique Ribeiro-
Implement unique and value_counts for SeriesGroupBy (GH#5358) Scott Sievert
Add sizeof definition for pyarrow tables and columns (GH#5522) Richard J Zamora
Enable row-group task partitioning in pyarrow-based read_parquet (GH#5508) Richard J Zamora
Removes npartitions=’auto’ from dd.merge docstring (GH#5531) James Bourbeau
Apply enforce error message shows non-overlapping columns. (GH#5530) Tom Augspurger
Optimize meta_nonempty for repetitive dtypes (GH#5553) Petio Petrov
Remove import of dask_cudf, which is now a part of cudf (GH#5568) Mads R. B. Kristensen
Documentation¶
Make capitalization more consistent in FAQ docs (GH#5512) Matthew Rocklin
Add CONTRIBUTING.md (GH#5513) Jacob Tomlinson
Document optional dependencies (GH#5456) Prithvi MK
Update helm chart docs to reflect new chart repo (GH#5539) Jacob Tomlinson
Add Resampler to API docs (GH#5551) James Bourbeau
Add adaptive deployments screencast [skip ci] (GH#5566) Matthew Rocklin
2.6.0 / 2019-10-15¶
Core¶
Call
ensure_dict
on graphs before enteringtoolz.merge
(GH#5486) Matthew RocklinConsolidating hash dispatch functions (GH#5476) Richard J Zamora
DataFrame¶
Support Python 3.5 in Parquet code (GH#5491) Ben Zaitlen
Avoid identity check in
warn_dtype_mismatch
(GH#5489) Tom AugspurgerEnable unused groupby tests (GH#3480) Jörg Dietrich
Remove old parquet and bcolz dataframe optimizations (GH#5484) Matthew Rocklin
Add getitem optimization for
read_parquet
(GH#5453) Tom AugspurgerUse
_constructor_sliced
method to determine Series type (GH#5480) Richard J ZamoraFix map(series) for unsorted base series index (GH#5459) Justin Waugh
Fix
KeyError
with Groupby label (GH#5467) Ryan Nazareth
Documentation¶
Use Zoom meeting instead of appear.in (GH#5494) Matthew Rocklin
Update SSH docs to include
SSHCluster
(GH#5482) Matthew RocklinUpdate “Why Dask?” page (GH#5473) Matthew Rocklin
2.5.2 / 2019-10-04¶
Array¶
Correct chunk size logic for asymmetric overlaps (GH#5449) Ben Jeffery
Make da.unify_chunks public API (GH#5443) Matthew Rocklin
DataFrame¶
Fix dask.dataframe.fillna handling of Scalar object (GH#5463) Zhenqing Li
Documentation¶
Remove boxes in Spark comparison page (GH#5445) Matthew Rocklin
Update cloud documentation (GH#5444) Matthew Rocklin
2.5.0 / 2019-09-27¶
Core¶
Add sentinel no_default to get_dependencies task (GH#5420) James Bourbeau
Update fsspec version (GH#5415) Matthew Rocklin
DataFrame¶
Add option to not check meta in dd.from_delayed (GH#5436) Christopher J. Wright
Fix test_timeseries_nulls_in_schema failures with pyarrow master (GH#5421) Richard J Zamora
Reduce read_metadata output size in pyarrow/parquet (GH#5391) Richard J Zamora
Test numeric edge case for repartition with npartitions. (GH#5433) amerkel2
Unxfail pandas-datareader test (GH#5430) Tom Augspurger
Add DataFrame.pop implementation (GH#5422) Matthew Rocklin
Enable merge/set_index for cudf-based dataframes with cupy
values
(GH#5322) Richard J Zamoradrop_duplicates support for positional subset parameter (GH#5410) Wes Roach
Documentation¶
Add screencasts to array, bag, dataframe, delayed, futures and setup (GH#5429) (GH#5424) Matthew Rocklin
Fix delimeter parsing documentation (GH#5428) Mahmut Bulut
Update overview image (GH#5404) James Bourbeau
2.4.0 / 2019-09-13¶
Array¶
Adds explicit
h5py.File
mode (GH#5390) James BourbeauProvides method to compute unknown array chunks sizes (GH#5312) Scott Sievert
Ignore runtime warning in Array
compute_meta
(GH#5356) estebanagAdd
_meta
toArray.__dask_postpersist__
(GH#5353) Benoit BovyFixup
da.asarray
andda.asanyarray
for datetime64 dtype and xarray objects (GH#5334) Stephan HoyerAdd shape implementation (GH#5293) Tom Augspurger
Add chunktype to array text repr (GH#5289) James Bourbeau
Array.random.choice: handle array-like non-arrays (GH#5283) Gabe Joseph
Core¶
Fix
funcname
when vectorized func has no__name__
(GH#5399) James BourbeauTruncate
funcname
to avoid long key names (GH#5383) Matthew RocklinAdd support for
numpy.vectorize
infuncname
(GH#5396) James BourbeauFixed HDFS upstream test (GH#5395) Tom Augspurger
Support numbers and None in
parse_bytes
/timedelta
(GH#5384) Matthew RocklinFix tokenizing of subindexes on memmapped numpy arrays (GH#5351) Henry Pinkard
Upstream fixups (GH#5300) Tom Augspurger
DataFrame¶
Allow pandas to cast type of statistics (GH#5402) Richard J Zamora
Preserve index dtype after applying
dd.pivot_table
(GH#5385) therhaagImplement explode for Series and DataFrame (GH#5381) Arpit Solanki
set_index
on categorical fails with less categories than partitions (GH#5354) Oliver HofkensSupport output to a single CSV file (GH#5304) Hongjiu Zhang
Add
groupby().transform()
(GH#5327) Oliver HofkensAdding filter kwarg to pyarrow dataset call (GH#5348) Richard J Zamora
Implement and check compression defaults for parquet (GH#5335) Sarah Bird
Pass sqlalchemy params to delayed objects (GH#5332) Arpit Solanki
Fixing schema handling in arrow-parquet (GH#5307) Richard J Zamora
Add support for DF and Series
groupby().idxmin/max()
(GH#5273) Oliver HofkensAdd correlation calculation and add test (GH#5296) Ben Zaitlen
Documentation¶
Minor edits to Array chunk documentation (GH#5372) Scott Sievert
Add methods to API docs (GH#5387) Tom Augspurger
Add namespacing to configuration example (GH#5374) Matthew Rocklin
Add get_task_stream and profile to the diagnostics page (GH#5375) Matthew Rocklin
Add best practice to load data with Dask (GH#5369) Matthew Rocklin
Add threads and processes note to the best practices (GH#5340) Matthew Rocklin
Update cuDF links (GH#5328) James Bourbeau
Fixed small typo with parentheses placement (GH#5311) Eugene Huang
Update link in reshape docstring (GH#5297) James Bourbeau
2.3.0 / 2019-08-16¶
Array¶
Raise exception when
from_array
is given a dask array (GH#5280) David HoeseAvoid adjusting gufunc’s meta dtype twice (GH#5274) Peter Andreas Entschev
Add
meta=
keyword to map_blocks and add test with sparse (GH#5269) Matthew RocklinAdd rollaxis and moveaxis (GH#4822) Tobias de Jong
Always increment old chunk index (GH#5256) James Bourbeau
Shuffle dask array (GH#3901) Tom Augspurger
Fix ordering when indexing a dask array with a bool dask array (GH#5151) James Bourbeau
Bag¶
Add workaround for memory leaks in bag generators (GH#5208) Marco Neumann
Core¶
Set strict xfail option (GH#5220) James Bourbeau
test-upstream (GH#5267) Tom Augspurger
Fixed HDFS CI failure (GH#5234) Tom Augspurger
Ensure parquet tests are skipped if fastparquet and pyarrow not installed (GH#5217) James Bourbeau
Add fsspec to readthedocs (GH#5207) Matthew Rocklin
Bump NumPy and Pandas to 1.17 and 0.25 in CI test (GH#5179) John A Kirkham
DataFrame¶
Fix
DataFrame.query
docstring (incorrect numexpr API) (GH#5271) Doug DavisParquet metadata-handling improvements (GH#5218) Richard J Zamora
Improve messaging around sorted parquet columns for index (GH#5265) Martin Durant
Add
rearrange_by_divisions
andset_index
support for cudf (GH#5205) Richard J ZamoraFix
groupby.std()
with integer colum names (GH#5096) Nicolas HugGeneralize
hash_pandas_object
to work for non-pandas backends (GH#5184) GALI PREM SAGARAdd rolling cov (GH#5154) Ivars Geidans
Add columns argument in drop function (GH#5223) Henrique Ribeiro
Documentation¶
Update institutional FAQ doc (GH#5277) Matthew Rocklin
Add draft of institutional FAQ (GH#5214) Matthew Rocklin
Make boxes for dask-spark page (GH#5249) Martin Durant
Add motivation for shuffle docs (GH#5213) Matthew Rocklin
Fix links and API entries for best-practices (GH#5246) Martin Durant
Remove “bytes” (internal data ingestion) doc page (GH#5242) Martin Durant
Redirect from our local distributed page to distributed.dask.org (GH#5248) Matthew Rocklin
Cleanup API page (GH#5247) Matthew Rocklin
Remove excess endlines from install docs (GH#5243) Matthew Rocklin
Remove item list in phases of computation doc (GH#5245) Martin Durant
Remove custom graphs from the TOC sidebar (GH#5241) Matthew Rocklin
Remove experimental status of custom collections (GH#5236) James Bourbeau
Adds table of contents to Why Dask? (GH#5244) James Bourbeau
Moves bag overview to top-level bag page (GH#5240) James Bourbeau
Remove use-cases in favor of stories.dask.org (GH#5238) Matthew Rocklin
Removes redundant TOC information in index.rst (GH#5235) James Bourbeau
Elevate dashboard in distributed diagnostics documentation (GH#5239) Martin Durant
Updates “add” layer in HLG docs example (GH#5237) James Bourbeau
Update GUFunc documentation (GH#5232) Matthew Rocklin
2.2.0 / 2019-08-01¶
Array¶
Use da.from_array(…, asarray=False) if input follows NEP-18 (GH#5074) Matthew Rocklin
Add missing attributes to from_array documentation (GH#5108) Peter Andreas Entschev
Fix meta computation for some reduction functions (GH#5035) Peter Andreas Entschev
Raise informative error in to_zarr if unknown chunks (GH#5148) James Bourbeau
Remove invalid pad tests (GH#5122) Tom Augspurger
Ignore NumPy warnings in compute_meta (GH#5103) Peter Andreas Entschev
Fix kurtosis calc for single dimension input array (GH#5177) @andrethrill
Support Numpy 1.17 in tests (GH#5192) Matthew Rocklin
Bag¶
Supply pool to bag test to resolve intermittent failure (GH#5172) Tom Augspurger
Core¶
Base dask on fsspec (GH#5064) (GH#5121) Martin Durant
Various upstream compatibility fixes (GH#5056) Tom Augspurger
Make distributed tests optional again. (GH#5128) Elliott Sales de Andrade
Fix HDFS in dask (GH#5130) Martin Durant
Ignore some more invalid value warnings. (GH#5140) Elliott Sales de Andrade
DataFrame¶
Fix pd.MultiIndex size estimate (GH#5066) Brett Naul
Generalizing has_known_categories (GH#5090) GALI PREM SAGAR
Refactor Parquet engine (GH#4995) Richard J Zamora
Add divide method to series and dataframe (GH#5094) msbrown47
fix flaky partd test (GH#5111) Tom Augspurger
Adjust is_dataframe_like to adjust for value_counts change (GH#5143) Tom Augspurger
Generalize rolling windows to support non-Pandas dataframes (GH#5149) Nick Becker
Avoid unnecessary aggregation in pivot_table (GH#5173) Daniel Saxton
Add column names to apply_and_enforce error message (GH#5180) Matthew Rocklin
Add schema keyword argument to to_parquet (GH#5150) Sarah Bird
Allow fastparquet to handle gather_statistics=False for file lists (GH#5157) Richard J Zamora
Documentation¶
Adds NumFOCUS badge to the README (GH#5086) James Bourbeau
Document DataFrame.set_index computataion behavior Natalya Rapstine
Use pip install . instead of calling setup.py (GH#5139) Matthias Bussonier
Close user survey (GH#5147) Tom Augspurger
Fix Google Calendar meeting link (GH#5155) Loïc Estève
Add docker image customization example (GH#5171) James Bourbeau
Update remote-data-services after fsspec (GH#5170) Martin Durant
Fix typo in spark.rst (GH#5164) Xavier Holt
Update setup/python docs for async/await API (GH#5163) Matthew Rocklin
Update Local Storage HPC documentation (GH#5165) Matthew Rocklin
2.1.0 / 2019-07-08¶
Array¶
Add
recompute=
keyword tosvd_compressed
for lower-memory use (GH#5041) Matthew RocklinChange
__array_function__
implementation for backwards compatibility (GH#5043) Ralf GommersAdded
dtype
andshape
kwargs toapply_along_axis
(GH#3742) Davis BennettFix reduction with empty tuple axis (GH#5025) Peter Andreas Entschev
Drop size 0 arrays in
stack
(GH#4978) John A Kirkham
Core¶
Removes index keyword from pandas
to_parquet
call (GH#5075) James BourbeauFixes upstream dev CI build installation (GH#5072) James Bourbeau
Ensure scalar arrays are not rendered to SVG (GH#5058) Willi Rath
Environment creation overhaul (GH#5038) Tom Augspurger
s3fs, moto compatibility (GH#5033) Tom Augspurger
pytest 5.0 compat (GH#5027) Tom Augspurger
DataFrame¶
Fix
compute_meta
recursion in blockwise (GH#5048) Peter Andreas EntschevRemove hard dependency on pandas in
get_dummies
(GH#5057) GALI PREM SAGARCheck dtypes unchanged when using
DataFrame.assign
(GH#5047) asmith26Fix cumulative functions on tables with more than 1 partition (GH#5034) tshatrov
Handle non-divisible sizes in repartition (GH#5013) George Sakkis
Handles timestamp and
preserve_index
changes in pyarrow (GH#5018) Richard J ZamoraFix undefined
meta
forstr.split(expand=False)
(GH#5022) Brett NaulRemoved checks used for debugging
merge_asof
(GH#5011) Cody JohnsonDon’t use type when getting accessor in dataframes (GH#4992) Matthew Rocklin
Add
melt
as a method of Dask DataFrame (GH#4984) Dustin TindallAdds path-like support to
to_hdf
(GH#5003) James Bourbeau
Documentation¶
Point to latest K8s setup article in JupyterHub docs (GH#5065) Sean McKenna
Changes vizualize to visualize (GH#5061) David Brochart
Fix
from_sequence
typo in delayed best practices (GH#5045) James BourbeauAdd user survey link to docs (GH#5026) James Bourbeau
Fixes typo in optimization docs (GH#5015) James Bourbeau
Update community meeting information (GH#5006) Tom Augspurger
2.0.0 / 2019-06-25¶
Array¶
Support automatic chunking in da.indices (GH#4981) James Bourbeau
Err if there are no arrays to stack (GH#4975) John A Kirkham
Asymmetrical Array Overlap (GH#4863) Michael Eaton
Dispatch concatenate where possible within dask array (GH#4669) Hameer Abbasi
Fix tokenization of memmapped numpy arrays on different part of same file (GH#4931) Henry Pinkard
Preserve NumPy condition in da.asarray to preserve output shape (GH#4945) Alistair Miles
Expand foo_like_safe usage (GH#4946) Peter Andreas Entschev
Defer order/casting einsum parameters to NumPy implementation (GH#4914) Peter Andreas Entschev
Remove numpy warning in moment calculation (GH#4921) Matthew Rocklin
Fix meta_from_array to support Xarray test suite (GH#4938) Matthew Rocklin
Cache chunk boundaries for integer slicing (GH#4923) Bruce Merry
Drop size 0 arrays in concatenate (GH#4167) John A Kirkham
Raise ValueError if concatenate is given no arrays (GH#4927) John A Kirkham
Promote types in concatenate using _meta (GH#4925) John A Kirkham
Add chunk type to html repr in Dask array (GH#4895) Matthew Rocklin
- Add Dask Array._meta attribute (GH#4543) Peter Andreas Entschev
Fix _meta slicing of flexible types (GH#4912) Peter Andreas Entschev
Minor meta construction cleanup in concatenate (GH#4937) Peter Andreas Entschev
Further relax Array meta checks for Xarray (GH#4944) Matthew Rocklin
Support meta= keyword in da.from_delayed (GH#4972) Matthew Rocklin
Concatenate meta along axis (GH#4977) John A Kirkham
Use meta in stack (GH#4976) John A Kirkham
Move blockwise_meta to more general compute_meta function (GH#4954) Matthew Rocklin
Alias .partitions to .blocks attribute of dask arrays (GH#4853) Genevieve Buckley
Drop outdated numpy_compat functions (GH#4850) John A Kirkham
Allow da.eye to support arbitrary chunking sizes with chunks=’auto’ (GH#4834) Anderson Banihirwe
Fix CI warnings in dask.array tests (GH#4805) Tom Augspurger
Make map_blocks work with drop_axis + block_info (GH#4831) Bruce Merry
Add SVG image and table in Array._repr_html_ (GH#4794) Matthew Rocklin
ufunc: avoid __array_wrap__ in favor of __array_function__ (GH#4708) Peter Andreas Entschev
Ensure trivial padding returns the original array (GH#4990) John A Kirkham
Test
da.block
with 0-size arrays (GH#4991) John A Kirkham
Core¶
Quiet dependency installs in CI (GH#4960) Tom Augspurger
Raise on warnings in tests (GH#4916) Tom Augspurger
Add a diagnostics extra to setup.py (includes bokeh) (GH#4924) John A Kirkham
Overload HighLevelGraphs values method (GH#4918) James Bourbeau
Add __await__ method to Dask collections (GH#4901) Matthew Rocklin
Also ignore AttributeErrors which may occur if snappy (not python-snappy) is installed (GH#4908) Mark Bell
Canonicalize key names in config.rename (GH#4903) Ian Bolliger
Bump minimum partd to 0.3.10 (GH#4890) Tom Augspurger
Catch async def SyntaxError (GH#4836) James Bourbeau
catch IOError in ensure_file (GH#4806) Justin Poehnelt
Cleanup CI warnings (GH#4798) Tom Augspurger
Move distributed’s parse and format functions to dask.utils (GH#4793) Matthew Rocklin
Apply black formatting (GH#4983) James Bourbeau
Package license file in wheels (GH#4988) John A Kirkham
DataFrame¶
Add an optional partition_size parameter to repartition (GH#4416) George Sakkis
merge_asof and prefix_reduction (GH#4877) Cody Johnson
Allow dataframes to be indexed by dask arrays (GH#4882) Endre Mark Borza
Avoid deprecated message parameter in pytest.raises (GH#4962) James Bourbeau
Update test_to_records to test with lengths argument(GH#4515) asmith26
Remove pandas pinning in Dataframe accessors (GH#4955) Matthew Rocklin
Fix correlation of series with same names (GH#4934) Philipp S. Sommer
Map Dask Series to Dask Series (GH#4872) Justin Waugh
Add groupby Covariance/Correlation (GH#4889) Ben Zaitlen
keep index name with to_datetime (GH#4905) Ian Bolliger
Add Parallel variance computation for dataframes (GH#4865) Ksenia Bobrova
Add divmod implementation to arrays and dataframes (GH#4884) Henrique Ribeiro
Add documentation for dataframe reshape methods (GH#4896) tpanza
Avoid use of pandas.compat (GH#4881) Tom Augspurger
Added accessor registration for Series, DataFrame, and Index (GH#4829) Tom Augspurger
Add read_function keyword to read_json (GH#4810) Richard J Zamora
Provide full type name in check_meta (GH#4819) Matthew Rocklin
Correctly estimate bytes per row in read_sql_table (GH#4807) Lijo Jose
Adding support of non-numeric data to describe() (GH#4791) Ksenia Bobrova
Scalars for extension dtypes. (GH#4459) Tom Augspurger
Call head before compute in dd.from_delayed (GH#4802) Matthew Rocklin
Add support for rolling operations with larger window that partition size in DataFrames with Time-based index (GH#4796) Jorge Pessoa
Update groupby-apply doc with warning (GH#4800) Tom Augspurger
Change groupby-ness tests in _maybe_slice (GH#4786) Ben Zaitlen
Add master best practices document (GH#4745) Matthew Rocklin
Add document for how Dask works with GPUs (GH#4792) Matthew Rocklin
Add cli API docs (GH#4788) James Bourbeau
Ensure concat output has coherent dtypes (GH#4692) Guillaume Lemaitre
Fixes pandas_datareader dependencies installation (GH#4989) James Bourbeau
Accept pathlib.Path as pattern in read_hdf (GH#3335) Jörg Dietrich
Documentation¶
Move CLI API docs to relavant pages (GH#4980) James Bourbeau
Add to_datetime function to dataframe API docs Matthew Rocklin
Add documentation entry for dask.array.ma.average (GH#4970) Bouwe Andela
Add bag.read_avro to bag API docs (GH#4969) James Bourbeau
Remove requirement to modify changelog (GH#4915) Matthew Rocklin
Add documentation about meta column order (GH#4887) Tom Augspurger
Add documentation note in DataFrame.shift (GH#4886) Tom Augspurger
Docs: Fix typo (GH#4868) Paweł Kordek
Put do/don’t into boxes for delayed best practice docs (GH#3821) Martin Durant
Doc fixups (GH#2528) Tom Augspurger
Add quansight to paid support doc section (GH#4838) Martin Durant
Add document for custom startup (GH#4833) Matthew Rocklin
Allow utils.derive_from to accept functions, apply across array (GH#4804) Martin Durant
Add “Avoid Large Partitions” section to best practices (GH#4808) Matthew Rocklin
Update URL for joblib to new website hosting their doc (GH#4816) Christian Hudon
1.2.2 / 2019-05-08¶
Array¶
Clarify regions kwarg to array.store (GH#4759) Martin Durant
Add dtype= parameter to da.random.randint (GH#4753) Matthew Rocklin
Use “row major” rather than “C order” in docstring (GH#4452) @asmith26
Normalize Xarray datasets to Dask arrays (GH#4756) Matthew Rocklin
Remove normed keyword in da.histogram (GH#4755) Matthew Rocklin
Bag¶
Add key argument to Bag.distinct (GH#4423) Daniel Severo
Core¶
Add core dask config file (GH#4774) Matthew Rocklin
Add core dask config file to MANIFEST.in (GH#4780) James Bourbeau
Enabling glob with HTTP file-system (GH#3926) Martin Durant
HTTPFile.seek with whence=1 (GH#4751) Martin Durant
DataFrame¶
Remove explicit references to Pandas in dask.dataframe.groupby (GH#4778) Matthew Rocklin
Add support for group_keys kwarg in DataFrame.groupby() (GH#4771) Brian Chu
Describe doc (GH#4762) Martin Durant
Remove explicit pandas check in cumulative aggregations (GH#4765) Nick Becker
Added meta for read_json and test (GH#4588) Abhinav Ralhan
Add test for dtype casting (GH#4760) Martin Durant
Implement Series.str.split(expand=True) (GH#4744) Matthew Rocklin
Documentation¶
Tweaks to develop.rst from trying to run tests (GH#4772) Christian Hudon
Add document describing phases of computation (GH#4766) Matthew Rocklin
Point users to Dask-Yarn from spark documentation (GH#4770) Matthew Rocklin
Update images in delayed doc to remove labels (GH#4768) Martin Durant
Explain intermediate storage for dask arrays (GH#4025) John A Kirkham
Specify bash code-block in array best practices (GH#4764) James Bourbeau
Add array best practices doc (GH#4705) Matthew Rocklin
Update optimization docs now that cull is not automatic (GH#4752) Matthew Rocklin
1.2.1 / 2019-04-29¶
Array¶
Fix map_blocks with block_info and broadcasting (GH#4737) Bruce Merry
Make ‘minlength’ keyword argument optional in da.bincount (GH#4684) Genevieve Buckley
Add support for map_blocks with no array arguments (GH#4713) Bruce Merry
Add dask.array.trace (GH#4717) Danilo Horta
Add sizeof support for cupy.ndarray (GH#4715) Peter Andreas Entschev
Add name kwarg to from_zarr (GH#4663) Michael Eaton
Add chunks=’auto’ to from_array (GH#4704) Matthew Rocklin
Raise TypeError if dask array is given as shape for da.ones, zeros, empty or full (GH#4707) Genevieve Buckley
Add TileDB backend (GH#4679) Isaiah Norton
Core¶
Delay long list arguments (GH#4735) Matthew Rocklin
Remove file “test” (GH#4710) James Bourbeau
Reenable development build, uses upstream libraries (GH#4696) Peter Andreas Entschev
Remove assertion in HighLevelGraph constructor (GH#4699) Matthew Rocklin
DataFrame¶
Change cum-aggregation last-nonnull-value algorithm (GH#4736) Nick Becker
Refactor array.percentile and dataframe.quantile to use t-digest (GH#4677) Janne Vuorela
Allow naive concatenation of sorted dataframes (GH#4725) Matthew Rocklin
Remove hard pandas dependency for melt by using methodcaller (GH#4719) Nick Becker
Add Dataframe.replace (GH#4714) Matthew Rocklin
Add ‘threshold’ parameter to pd.DataFrame.dropna (GH#4625) Nathan Matare
Documentation¶
Add warning about derived docstrings early in the docstring (GH#4716) Matthew Rocklin
Create dataframe best practices doc (GH#4703) Matthew Rocklin
Uncomment dask_sphinx_theme (GH#4728) James Bourbeau
Fix minor typo fix in a Queue/fire_and_forget example (GH#4709) Matthew Rocklin
Update from_pandas docstring to match signature (GH#4698) James Bourbeau
1.2.0 / 2019-04-12¶
Array¶
Fixed mean() and moment() on sparse arrays (GH#4525) Peter Andreas Entschev
Add test for NEP-18. (GH#4675) Hameer Abbasi
Allow None to say “no chunking” in normalize_chunks (GH#4656) Matthew Rocklin
Fix limit value in auto_chunks (GH#4645) Matthew Rocklin
Core¶
Updated diagnostic bokeh test for compatibility with bokeh>=1.1.0 (GH#4680) Philipp Rudiger
Adjusts codecov’s target/threshold, disable patch (GH#4671) Peter Andreas Entschev
Always start with empty http buffer, not None (GH#4673) Martin Durant
DataFrame¶
Propagate index dtype and name when create dask dataframe from array (GH#4686) Henrique Ribeiro
Clean up and document rearrange_column_by_tasks (GH#4674) Matthew Rocklin
Mark some parquet tests xfail (GH#4667) Peter Andreas Entschev
Fix parquet breakages with arrow 0.13.0 (GH#4668) Martin Durant
Allow sample to be False when reading CSV from a remote URL (GH#4634) Ian Rose
Fix timezone metadata inference on parquet load (GH#4655) Martin Durant
Use is_dataframe/index_like in dd.utils (GH#4657) Matthew Rocklin
Add min_count parameter to groupby sum method (GH#4648) Henrique Ribeiro
Correct quantile to handle unsorted quantiles (GH#4650) gregrf
Documentation¶
Add delayed extra dependencies to install docs (GH#4660) James Bourbeau
1.1.5 / 2019-03-29¶
Array¶
Ensure that we use the dtype keyword in normalize_chunks (GH#4646) Matthew Rocklin
Core¶
Use recursive glob in LocalFileSystem (GH#4186) Brett Naul
Avoid YAML deprecation (GH#4603)
Fix CI and add set -e (GH#4605) James Bourbeau
Support builtin sequence types in dask.visualize (GH#4602)
unpack/repack orderedDict (GH#4623) Justin Poehnelt
Add da.random.randint to API docs (GH#4628) James Bourbeau
Add zarr to CI environment (GH#4604) James Bourbeau
Enable codecov (GH#4631) Peter Andreas Entschev
DataFrame¶
Support setting the index (GH#4565)
DataFrame.itertuples accepts index, name kwargs (GH#4593) Dan O’Donovan
Support non-Pandas series in dd.Series.unique (GH#4599) Ben Zaitlen
Replace use of explicit type check with ._is_partition_type predicate (GH#4533)
Remove additional pandas warnings in tests (GH#4576)
Check object for name/dtype attributes rather than type (GH#4606)
Fixing warning from setting categorical codes to floats (GH#4624) Julia Signell
Fix renaming on index to_frame method (GH#4498) Henrique Ribeiro
Fix divisions when joining two single-partition dataframes (GH#4636) Justin Waugh
Warn if partitions overlap in compute_divisions (GH#4600) Brian Chu
Give informative meta= warning (GH#4637) Matthew Rocklin
Add informative error message to Series.__getitem__ (GH#4638) Matthew Rocklin
Add clear exception message when using index or index_col in read_csv (GH#4651) Álvaro Abella Bascarán
Documentation¶
Add documentation for custom groupby aggregations (GH#4571)
Docs dataframe joins (GH#4569)
Specify fork-based contributions (GH#4619) James Bourbeau
correct to_parquet example in docs (GH#4641) Aaron Fowles
Update and secure several references (GH#4649) Søren Fuglede Jørgensen
1.1.4 / 2019-03-08¶
Array¶
Use mask selection in compress (GH#4548) John A Kirkham
Use asarray in extract (GH#4549) John A Kirkham
Use correct dtype when test concatenation. (GH#4539) Elliott Sales de Andrade
Fix CuPy tests or properly marks as xfail (GH#4564) Peter Andreas Entschev
Core¶
Fix local scheduler callback to deal with custom caching (GH#4542) Yu Feng
Use parse_bytes in read_bytes(sample=…) (GH#4554) Matthew Rocklin
DataFrame¶
Fix up groupby-standard deviation again on object dtype keys (GH#4541) Matthew Rocklin
TST/CI: Updates for pandas 0.24.1 (GH#4551) Tom Augspurger
Add ability to control number of unique elements in timeseries (GH#4557) Matthew Rocklin
Add support in read_csv for parameter skiprows for other iterables (GH#4560) @JulianWgs
Documentation¶
DataFrame to Array conversion and unknown chunks (GH#4516) Scott Sievert
Add docs for random array creation (GH#4566) Matthew Rocklin
Fix typo in docstring (GH#4572) Shyam Saladi
1.1.3 / 2019-03-01¶
Array¶
Modify mean chunk functions to return dicts rather than arrays (GH#4513) Matthew Rocklin
Change sparse installation in CI for NumPy/Python2 compatibility (GH#4537) Matthew Rocklin
DataFrame¶
Make merge dispatchable on pandas/other dataframe types (GH#4522) Matthew Rocklin
read_sql_table - datetime index fix and index type checking (GH#4474) Joe Corbett
Use generalized form of index checking (is_index_like) (GH#4531) Ben Zaitlen
Add tests for groupby reductions with object dtypes (GH#4535) Matthew Rocklin
Fixes #4467 : Updates time_series for pandas deprecation (GH#4530) @HSR05
Documentation¶
Add missing method to documentation index (GH#4528) Bart Broere
1.1.2 / 2019-02-25¶
Array¶
Fix another unicode/mixed-type edge case in normalize_array (GH#4489) Marco Neumann
Add dask.array.diagonal (GH#4431) Danilo Horta
Modify moment chunk functions to return dicts (GH#4519) Peter Andreas Entschev
Bag¶
Ensure that bag.from_sequence always includes at least one partition (GH#4475) Anderson Banihirwe
Implement out_type for bag.fold (GH#4502) Matthew Rocklin
Remove map from bag keynames (GH#4500) Matthew Rocklin
Avoid itertools.repeat in map_partitions (GH#4507) Matthew Rocklin
DataFrame¶
Fix relative path parsing on windows when using fastparquet (GH#4445) Janne Vuorela
Fix bug in pyarrow and hdfs (GH#4453) (GH#4455) Michał Jastrzębski
df getitem with integer slices is not implemented (GH#4466) Jim Crist
Replace cudf-specific code with dask-cudf import (GH#4470) Matthew Rocklin
Avoid groupby.agg(callable) in groupby-var (GH#4482) Matthew Rocklin
Consider uint types as numerical in check_meta (GH#4485) Marco Neumann
Fix some typos in groupby comments (GH#4494) Daniel Saxton
Add error message around set_index(inplace=True) (GH#4501) Matthew Rocklin
meta_nonempty works with categorical index (GH#4505) Jim Crist
Add module name to expected meta error message (GH#4499) Matthew Rocklin
Propogate index metadata if not specified (GH#4509) Jim Crist
Documentation¶
Update docs to use
from_zarr
(GH#4472) John A KirkhamDOC: add section of Using Other S3-Compatible Services for remote-data-services (GH#4405) Aploium
Fix header level of section in changelog (GH#4483) Bruce Merry
Add quotes to pip install [skip-ci] (GH#4508) James Bourbeau
Core¶
Extend started_cbs AFTER state is initialized (GH#4460) Marco Neumann
Fix bug in HTTPFile._fetch_range with headers (GH#4479) (GH#4480) Ross Petchler
Repeat optimize_blockwise for diamond fusion (GH#4492) Matthew Rocklin
1.1.1 / 2019-01-31¶
Array¶
Add support for cupy.einsum (GH#4402) Johnnie Gray
Provide byte size in chunks keyword (GH#4434) Adam Beberg
Raise more informative error for histogram bins and range (GH#4430) James Bourbeau
DataFrame¶
Lazily register more cudf functions and move to backends file (GH#4396) Matthew Rocklin
rearrange_by_column: ensure that shuffle arg defaults to ‘disk’ if it’s None in dask.config (GH#4414) George Sakkis
Implement filters for _read_pyarrow (GH#4415) George Sakkis
Avoid checking against types in is_dataframe_like (GH#4418) Matthew Rocklin
Pass username as ‘user’ when using pyarrow (GH#4438) Roma Sokolov
Delayed¶
Fix DelayedAttr return value (GH#4440) Matthew Rocklin
Documentation¶
Use SVG for pipeline graphic (GH#4406) John A Kirkham
Add doctest-modules to py.test documentation (GH#4427) Daniel Severo
Core¶
Work around psutil 5.5.0 not allowing pickling Process objects Janne Vuorela
1.1.0 / 2019-01-18¶
Array¶
Fix the average function when there is a masked array (GH#4236) Damien Garaud
Add allow_unknown_chunksizes to hstack and vstack (GH#4287) Paul Vecchio
Fix tensordot for 27+ dimensions (GH#4304) Johnnie Gray
Fixed block_info with axes. (GH#4301) Tom Augspurger
Use safe_wraps for matmul (GH#4346) Mark Harfouche
Use chunks=”auto” in array creation routines (GH#4354) Matthew Rocklin
Fix np.matmul in dask.array.Array.__array_ufunc__ (GH#4363) Stephan Hoyer
COMPAT: Re-enable multifield copy->view change (GH#4357) Diane Trout
Calling np.dtype on a delayed object works (GH#4387) Jim Crist
Rework normalize_array for numpy data (GH#4312) Marco Neumann
DataFrame¶
Add fill_value support for series comparisons (GH#4250) James Bourbeau
Add schema name in read_sql_table for empty tables (GH#4268) Mina Farid
Adjust check for bad chunks in map_blocks (GH#4308) Tom Augspurger
Use atop fusion in dask dataframe (GH#4229) Matthew Rocklin
Use parallel_types() in from_pandas (GH#4331) Matthew Rocklin
Change DataFrame._repr_data to method (GH#4330) Matthew Rocklin
Install pyarrow fastparquet for Appveyor (GH#4338) Gábor Lipták
Remove explicit pandas checks and provide cudf lazy registration (GH#4359) Matthew Rocklin
Replace isinstance(…, pandas) with is_dataframe_like (GH#4375) Matthew Rocklin
ENH: Support 3rd-party ExtensionArrays (GH#4379) Tom Augspurger
Pandas 0.24.0 compat (GH#4374) Tom Augspurger
Documentation¶
Fix link to ‘map_blocks’ function in array api docs (GH#4258) David Hoese
Add a paragraph on Dask-Yarn in the cloud docs (GH#4260) Jim Crist
Copy edit documentation (GH#4267), (GH#4263), (GH#4262), (GH#4277), (GH#4271), (GH#4279), (GH#4265), (GH#4295), (GH#4293), (GH#4296), (GH#4302), (GH#4306), (GH#4318), (GH#4314), (GH#4309), (GH#4317), (GH#4326), (GH#4325), (GH#4322), (GH#4332), (GH#4333), Miguel Farrajota
Doc: Update array-api.rst (GH#4259) (GH#4282) Prabakaran Kumaresshan
Update hpc doc (GH#4266) Guillaume Eynard-Bontemps
Doc: Replace from_avro with read_avro in documents (GH#4313) Prabakaran Kumaresshan
Remove reference to “get” scheduler functions in docs (GH#4350) Matthew Rocklin
Fix typo in docstring (GH#4376) Daniel Saxton
Added documentation for dask.dataframe.merge (GH#4382) Jendrik Jördening
Core¶
Avoid recursion in dask.core.get (GH#4219) Matthew Rocklin
Remove verbose flag from pytest setup.cfg (GH#4281) Matthew Rocklin
Support Pytest 4.0 by specifying marks explicitly (GH#4280) Takahiro Kojima
Add High Level Graphs (GH#4092) Matthew Rocklin
Fix SerializableLock locked and acquire methods (GH#4294) Stephan Hoyer
Pin boto3 to earlier version in tests to avoid moto conflict (GH#4276) Martin Durant
Treat None as missing in config when updating (GH#4324) Matthew Rocklin
Update Appveyor to Python 3.6 (GH#4337) Gábor Lipták
Use parse_bytes more liberally in dask.dataframe/bytes/bag (GH#4339) Matthew Rocklin
Add a better error message when cloudpickle is missing (GH#4342) Mark Harfouche
Support pool= keyword argument in threaded/multiprocessing get functions (GH#4351) Matthew Rocklin
Allow updates from arbitrary Mappings in config.update, not only dicts. (GH#4356) Stuart Berg
Move dask/array/top.py code to dask/blockwise.py (GH#4348) Matthew Rocklin
Add has_parallel_type (GH#4395) Matthew Rocklin
CI: Update Appveyor (GH#4381) Tom Augspurger
1.0.0 / 2018-11-28¶
Array¶
Add nancumsum/nancumprod unit tests (GH#4215) Guido Imperiale
DataFrame¶
Add index to to_dask_dataframe docstring (GH#4232) James Bourbeau
Text and fix when appending categoricals with fastparquet (GH#4245) Martin Durant
Don’t reread metadata when passing ParquetFile to read_parquet (GH#4247) Martin Durant
Documentation¶
Core¶
Avoid a few warnings (GH#4223) Matthew Rocklin
Remove dask.store module (GH#4221) Matthew Rocklin
Remove AUTHORS.md Jim Crist
0.20.2 / 2018-11-15¶
Array¶
Avoid fusing dependencies of atop reductions (GH#4207) Matthew Rocklin
Dataframe¶
Improve memory footprint for dataframe correlation (GH#4193) Damien Garaud
Add empty DataFrame check to boundary_slice (GH#4212) James Bourbeau
Documentation¶
Copy edit documentation (GH#4197) (GH#4204) (GH#4198) (GH#4199) (GH#4200) (GH#4202) (GH#4209) Miguel Farrajota
Add stats module namespace (GH#4206) James Bourbeau
Fix link in dataframe documentation (GH#4208) James Bourbeau
0.20.1 / 2018-11-09¶
Array¶
Only allocate the result space in wrapped_pad_func (GH#4153) John A Kirkham
Generalize expand_pad_width to expand_pad_value (GH#4150) John A Kirkham
Test da.pad with 2D linear_ramp case (GH#4162) John A Kirkham
Rewrite Dask Array’s pad to add only new chunks (GH#4152) John A Kirkham
Validate index inputs to atop (GH#4182) Matthew Rocklin
Core¶
Dask.config set and get normalize underscores and hyphens (GH#4143) James Bourbeau
Only subs on core collections, not subclasses (GH#4159) Matthew Rocklin
Add block_size=0 option to HTTPFileSystem. (GH#4171) Martin Durant
Add traverse support for dataclasses (GH#4165) Armin Berres
Avoid optimization on sharedicts without dependencies (GH#4181) Matthew Rocklin
Update the pytest version for TravisCI (GH#4189) Damien Garaud
Use key_split rather than funcname in visualize names (GH#4160) Matthew Rocklin
Dataframe¶
Add fix for DataFrame.__setitem__ for index (GH#4151) Anderson Banihirwe
Fix column choice when passing list of files to fastparquet (GH#4174) Martin Durant
Pass engine_kwargs from read_sql_table to sqlalchemy (GH#4187) Damien Garaud
Documentation¶
Fix documentation in Delayed best practices example that returned an empty list (GH#4147) Jonathan Fraine
Copy edit documentation (GH#4164) (GH#4175) (GH#4185) (GH#4192) (GH#4191) (GH#4190) (GH#4180) Miguel Farrajota
Fix typo in docstring (GH#4183) Carlos Valiente
0.20.0 / 2018-10-26¶
Array¶
Fuse Atop operations (GH#3998), (GH#4081) Matthew Rocklin
Support da.asanyarray on dask dataframes (GH#4080) Matthew Rocklin
Remove unnecessary endianness check in datetime test (GH#4113) Elliott Sales de Andrade
Set name=False in array foo_like functions (GH#4116) Matthew Rocklin
Remove dask.array.ghost module (GH#4121) Matthew Rocklin
Fix use of getargspec in dask array (GH#4125) Stephan Hoyer
Adds dask.array.invert (GH#4127), (GH#4131) Anderson Banihirwe
Raise informative error on arg-reduction on unknown chunksize (GH#4128), (GH#4135) Matthew Rocklin
Normalize reversed slices in dask array (GH#4126) Matthew Rocklin
Bag¶
Add bag.to_avro (GH#4076) Martin Durant
Core¶
Pull num_workers from config.get (GH#4086), (GH#4093) James Bourbeau
Fix invalid escape sequences with raw strings (GH#4112) Elliott Sales de Andrade
Raise an error on the use of the get= keyword and set_options (GH#4077) Matthew Rocklin
Add import for Azure DataLake storage, and add docs (GH#4132) Martin Durant
Avoid collections.Mapping/Sequence (GH#4138) Matthew Rocklin
Dataframe¶
Include index keyword in to_dask_dataframe (GH#4071) Matthew Rocklin
Implement min_count for the DataFrame methods sum and prod (GH#4090) Bart Broere
Remove pandas warnings in concat (GH#4095) Matthew Rocklin
DataFrame.to_csv header option to only output headers in the first chunk (GH#3909) Rahul Vaidya
Remove Series.to_parquet (GH#4104) Justin Dennison
Avoid warnings and deprecated pandas methods (GH#4115) Matthew Rocklin
Swap ‘old’ and ‘previous’ when reporting append error (GH#4130) Martin Durant
Documentation¶
Copy edit documentation (GH#4073), (GH#4074), (GH#4094), (GH#4097), (GH#4107), (GH#4124), (GH#4133), (GH#4139) Miguel Farrajota
Fix typo in code example (GH#4089) Antonino Ingargiola
Quick description for gcsfs (GH#4109) Martin Durant
Fixed typo in docstrings of read_sql_table method (GH#4114) TakaakiFuruse
Make target directories in redirects if they don’t exist (GH#4136) Matthew Rocklin
0.19.4 / 2018-10-09¶
Array¶
Implement
apply_gufunc(..., axes=..., keepdims=...)
(GH#3985) Markus Gonser
Bag¶
Fix typo in datasets.make_people (GH#4069) Matthew Rocklin
Dataframe¶
Added percentiles options for dask.dataframe.describe method (GH#4067) Zhenqing Li
Add DataFrame.partitions accessor similar to Array.blocks (GH#4066) Matthew Rocklin
Core¶
Pass get functions and Clients through scheduler keyword (GH#4062) Matthew Rocklin
Documentation¶
Fix Typo on hpc example. (missing = in kwarg). (GH#4068) Matthias Bussonier
Extensive copy-editing: (GH#4065), (GH#4064), (GH#4063) Miguel Farrajota
0.19.3 / 2018-10-05¶
Array¶
Make da.RandomState extensible to other modules (GH#4041) Matthew Rocklin
Support unknown dims in ravel no-op case (GH#4055) Jim Crist
Add basic infrastructure for cupy (GH#4019) Matthew Rocklin
Avoid asarray and lock arguments for from_array(getitem) (GH#4044) Matthew Rocklin
Move local imports in corrcoef to global imports (GH#4030) John A Kirkham
Move local indices import to global import (GH#4029) John A Kirkham
Fix-up Dask Array’s fromfunction w.r.t. dtype and kwargs (GH#4028) John A Kirkham
Don’t use dummy expansion for trim_internal in overlapped (GH#3964) Mark Harfouche
Add unravel_index (GH#3958) John A Kirkham
Bag¶
Sort result in Bag.frequencies (GH#4033) Matthew Rocklin
Add support for npartitions=1 edge case in groupby (GH#4050) James Bourbeau
Add new random dataset for people (GH#4018) Matthew Rocklin
Improve performance of bag.read_text on small files (GH#4013) Eric Wolak
Add bag.read_avro (GH#4000) (GH#4007) Martin Durant
Dataframe¶
Added an
index
parameter todask.dataframe.from_dask_array()
for creating a dask DataFrame from a dask Array with a given index. (GH#3991) Tom AugspurgerImprove sub-classability of dask dataframe (GH#4015) Matthew Rocklin
fuse_subgraphs works without normal fuse (GH#4042) Jim Crist
Make path for reading many parquet files without prescan (GH#3978) Martin Durant
Index in dd.from_dask_array (GH#3991) Tom Augspurger
Making skiprows accept lists (GH#3975) Julia Signell
Fail early in fastparquet read for nonexistent column (GH#3989) Martin Durant
Core¶
Add support for npartitions=1 edge case in groupby (GH#4050) James Bourbeau
Automatically wrap large arguments with dask.delayed in map_blocks/partitions (GH#4002) Matthew Rocklin
Make multiprocessing context configurable (GH#3763) Itamar Turner-Trauring
Documentation¶
Extensive copy-editing (GH#4049), (GH#4034), (GH#4031), (GH#4020), (GH#4021), (GH#4022), (GH#4023), (GH#4016), (GH#4017), (GH#4010), (GH#3997), (GH#3996), Miguel Farrajota
Update shuffle method selection docs (GH#4048) James Bourbeau
Remove docs/source/examples, point to examples.dask.org (GH#4014) Matthew Rocklin
Replace readthedocs links with dask.org (GH#4008) Matthew Rocklin
Updates DataFrame.to_hdf docstring for returned values (GH#3992) James Bourbeau
0.19.2 / 2018-09-17¶
Array¶
apply_gufunc
implements automatic infer of functions output dtypes (GH#3936) Markus GonserFix array histogram range error when array has nans (GH#3980) James Bourbeau
from_array: add @martindurant’s explaining of how hashing is done for an array. (GH#3965) Mark Harfouche
Support gradient with coordinate (GH#3949) Keisuke Fujii
Core¶
Fix use of has_keyword with partial in Python 2.7 (GH#3966) Mark Harfouche
Set pyarrow as default for HDFS (GH#3957) Matthew Rocklin
Documentation¶
Use dask_sphinx_theme (GH#3963) Matthew Rocklin
Use JupyterLab in Binder links from main page Matthew Rocklin
DOC: fixed sphinx syntax (GH#3960) Tom Augspurger
0.19.1 / 2018-09-06¶
Array¶
Don’t enforce dtype if result has no dtype (GH#3928) Matthew Rocklin
Fix NumPy issubtype deprecation warning (GH#3939) Bruce Merry
Fix arg reduction tokens to be unique with different arguments (GH#3955) Tobias de Jong
Coerce numpy integers to ints in slicing code (GH#3944) Yu Feng
Linalg.norm ndim along axis partial fix (GH#3933) Tobias de Jong
Dataframe¶
Deterministic DataFrame.set_index (GH#3867) George Sakkis
Fix divisions in read_parquet when dealing with filters #3831 #3930 (GH#3923) (GH#3931) @andrethrill
Fixing returning type in categorical.as_known (GH#3888) Sriharsha Hatwar
Fix DataFrame.assign for callables (GH#3919) Tom Augspurger
Include partitions with no width in repartition (GH#3941) Matthew Rocklin
Don’t constrict stage/k dtype in dataframe shuffle (GH#3942) Matthew Rocklin
Documentation¶
DOC: Add hint on how to render task graphs horizontally (GH#3922) Uwe Korn
Add try-now button to main landing page (GH#3924) Matthew Rocklin
0.19.0 / 2018-08-29¶
Array¶
Support coordinate in gradient (GH#3949) Keisuke Fujii
Fix argtopk split_every bug (GH#3810) Guido Imperiale
Ensure result computing dask.array.isnull() always gives a numpy array (GH#3825) Stephan Hoyer
Support concatenate for scipy.sparse in dask array (GH#3836) Matthew Rocklin
Fix argtopk on 32-bit systems. (GH#3823) Elliott Sales de Andrade
Normalize keys in rechunk (GH#3820) Matthew Rocklin
Allow shape of dask.array to be a numpy array (GH#3844) Mark Harfouche
Fix numpy deprecation warning on tuple indexing (GH#3851) Tobias de Jong
Rename ghost module to overlap (GH#3830) Robert Sare
Ensure copy preserves masked arrays (GH#3852) Tobias de Jong
DataFrame¶
Added
dtype
andsparse
keywords todask.dataframe.get_dummies()
(GH#3792) Tom AugspurgerAdded
dask.dataframe.to_dask_array()
for converting a Dask Series or DataFrame to a Dask Array, possibly with known chunk sizes (GH#3884) Tom AugspurgerChanged the behavior for
dask.array.asarray()
for dask dataframe and series inputs. Previously, the series was eagerly converted to an in-memory NumPy array before creating a dask array with known chunks sizes. This caused unexpectedly high memory usage. Now, no intermediate NumPy array is created, and a Dask array with unknown chunk sizes is returned (GH#3884) Tom AugspurgerDataFrame.iloc (GH#3805) Tom Augspurger
When reading multiple paths, expand globs. (GH#3828) Irina Truong
Added index column name after resample (GH#3833) Eric Bonfadini
Add (lazy) shape property to dataframe and series (GH#3212) Henrique Ribeiro
Rename to_csv keys for diagnostics (GH#3890) Matthew Rocklin
Match pandas warnings for concat sort (GH#3897) Tom Augspurger
Include filename in read_csv (GH#3908) Julia Signell
Core¶
Better error message on import when missing common dependencies (GH#3771) Danilo Horta
Add DASK_ROOT_CONFIG environment variable (GH#3849) Joe Hamman
Don’t cull in local scheduler, do cull in delayed (GH#3856) Jim Crist
Fix collections.abc deprecation warnings in Python 3.7.0 (GH#3876) Jan Margeta
Allow dot jpeg to xfail in visualize tests (GH#3896) Matthew Rocklin
Add Python 3.7 to travis.yml (GH#3894) Matthew Rocklin
Add expand_environment_variables to dask.config (GH#3893) Joe Hamman
Docs¶
Fix typo in import statement of diagnostics (GH#3826) John Mrziglod
fix of minor typos in landing page index.html (GH#3746) Christoph Moehl
Update delayed-custom.rst (GH#3850) Anderson Banihirwe
DOC: clarify delayed docstring (GH#3709) Scott Sievert
Add dask array normalize_chunks to documentation (GH#3878) Daniel Rothenberg
Docs: Fix link to snakeviz (GH#3900) Hans Moritz Günther
0.18.2 / 2018-07-23¶
Array¶
Reimplemented
argtopk
to make it release the GIL (GH#3610) Guido ImperialeDon’t overlap on non-overlapped dimensions in
map_overlap
(GH#3653) Matthew RocklinFix
linalg.tsqr
for dimensions of uncertain length (GH#3662) Jeremy ChenBreak apart uneven array-of-int slicing to separate chunks (GH#3648) Matthew Rocklin
Align auto chunks to provided chunks, rather than shape (GH#3679) Matthew Rocklin
Adds endpoint and retstep support for linspace (GH#3675) James Bourbeau
Implement
.blocks
accessor (GH#3689) Matthew RocklinAdd
block_info
keyword tomap_blocks
functions (GH#3686) Matthew RocklinSlice by dask array of ints (GH#3407) Guido Imperiale
Support
dtype
inarange
(GH#3722) Guido ImperialeFix
argtopk
with uneven chunks (GH#3720) Guido ImperialeRaise error when
replace=False
inda.choice
(GH#3765) James BourbeauUpdate chunks in
Array.__setitem__
(GH#3767) Itamar Turner-TrauringAdd a
chunksize
convenience property (GH#3777) Jacob TomlinsonFix and simplify array slicing behavior when
step < 0
(GH#3702) Ziyao WeiEnsure
to_zarr
withreturn_stored
True
returns a Dask Array (GH#3786) John A Kirkham
Bag¶
Add
last_endline
optional parameter into_textfiles
(GH#3745) George Sakkis
Dataframe¶
Add aggregate function for rolling objects (GH#3772) Gerome Pistre
Properly tokenize cumulative groupby aggregations (GH#3799) Cloves Almeida
Delayed¶
Add the
@
operator to the delayed objects (GH#3691) Mark HarfoucheAdd delayed best practices to documentation (GH#3737) Matthew Rocklin
Fix
@delayed
decorator for methods and add tests (GH#3757) Ziyao Wei
Core¶
Fix extra progressbar (GH#3669) Mike Neish
Allow tasks back onto ordering stack if they have one dependency (GH#3652) Matthew Rocklin
Prefer end-tasks with low numbers of dependencies when ordering (GH#3588) Tom Augspurger
Add
assert_eq
to top-level modules (GH#3726) Matthew RocklinTest that dask collections can hold
scipy.sparse
arrays (GH#3738) Matthew RocklinFix setup of lz4 decompression functions (GH#3782) Elliott Sales de Andrade
Add datasets module (GH#3780) Matthew Rocklin
0.18.1 / 2018-06-22¶
Array¶
from_array
now supports scalar types and nested lists/tuples in input, just like all numpy functions do; it also produces a simpler graph when the input is a plain ndarray (GH#3568) Guido ImperialeFix slicing of big arrays due to cumsum dtype bug (GH#3620) Marco Rossi
Add Dask Array implementation of pad (GH#3578) John A Kirkham
Fix array random API examples (GH#3625) James Bourbeau
Add average function to dask array (GH#3640) James Bourbeau
Tokenize ghost_internal with axes (GH#3643) Matthew Rocklin
Add outer for Dask Arrays (GH#3658) John A Kirkham
DataFrame¶
Add Index.to_series method (GH#3613) Henrique Ribeiro
Fix missing partition columns in pyarrow-parquet (GH#3636) Martin Durant
Core¶
Minor tweaks to CI (GH#3629) Guido Imperiale
Add back dask.utils.effective_get (GH#3642) Matthew Rocklin
DASK_CONFIG dictates config write location (GH#3621) Jim Crist
Replace ‘collections’ key in unpack_collections with unique key (GH#3632) Yu Feng
Avoid deepcopy in dask.config.set (GH#3649) Matthew Rocklin
0.18.0 / 2018-06-14¶
Array¶
Add to/from_zarr for Zarr-format datasets and arrays (GH#3460) Martin Durant
Experimental addition of generalized ufunc support,
apply_gufunc
,gufunc
, andas_gufunc
(GH#3109) (GH#3526) (GH#3539) Markus GonserAvoid unnecessary rechunking tasks (GH#3529) Matthew Rocklin
Compute dtypes at runtime for fft (GH#3511) Matthew Rocklin
Generate UUIDs for all da.store operations (GH#3540) Martin Durant
Correct internal dimension of Dask’s SVD (GH#3517) John A Kirkham
BUG: do not raise IndexError for identity slice in array.vindex (GH#3559) Scott Sievert
Adds isneginf and isposinf (GH#3581) John A Kirkham
Drop Dask Array’s learn module (GH#3580) John A Kirkham
added sfqr (short-and-fat) as a counterpart to tsqr… (GH#3575) Jeremy Chen
Allow 0-width chunks in dask.array.rechunk (GH#3591) Marc Pfister
Document Dask Array’s nan_to_num in public API (GH#3599) John A Kirkham
Show block example (GH#3601) John A Kirkham
Replace token= keyword with name= in map_blocks (GH#3597) Matthew Rocklin
Disable locking in to_zarr (needed for using to_zarr in a distributed context) (GH#3607) John A Kirkham
Support Zarr Arrays in to_zarr/from_zarr (GH#3561) John A Kirkham
Added recursion to array/linalg/tsqr to better manage the single core bottleneck (GH#3586) Jeremy Chan (GH#3396) Guido Imperiale
Dataframe¶
Add to/read_json (GH#3494) Martin Durant
Adds
index
to unsupported arguments forDataFrame.rename
method (GH#3522) James BourbeauAdds support to subset Dask DataFrame columns using
numpy.ndarray
,pandas.Series
, andpandas.Index
objects (GH#3536) James BourbeauRaise error if meta columns do not match dataframe (GH#3485) Christopher Ren
Add index to unsupprted argument for DataFrame.rename (GH#3522) James Bourbeau
Adds support for subsetting DataFrames with pandas Index/Series and numpy ndarrays (GH#3536) James Bourbeau
Dataframe sample method docstring fix (GH#3566) James Bourbeau
fixes dd.read_json to infer file compression (GH#3594) Matt Lee
Adds n to sample method (GH#3606) James Bourbeau
Add fastparquet ParquetFile object support (GH#3573) @andrethrill
Bag¶
Rename method= keyword to shuffle= in bag.groupby (GH#3470) Matthew Rocklin
Core¶
Replace get= keyword with scheduler= keyword (GH#3448) Matthew Rocklin
Add centralized dask.config module to handle configuration for all Dask subprojects (GH#3432) (GH#3513) (GH#3520) Matthew Rocklin
Read whole files fix regardless of header for HTTP (GH#3496) Martin Durant
Adds synchronous scheduler syntax to debugging docs (GH#3509) James Bourbeau
Replace dask.set_options with dask.config.set (GH#3502) Matthew Rocklin
Update sphinx readthedocs-theme (GH#3516) Matthew Rocklin
Introduce “auto” value for normalize_chunks (GH#3507) Matthew Rocklin
Fix check in configuration with env=None (GH#3562) Simon Perkins
Update sizeof definitions (GH#3582) Matthew Rocklin
Remove –verbose flag from travis-ci (GH#3477) Matthew Rocklin
Remove “da.random” from random array keys (GH#3604) Matthew Rocklin
0.17.5 / 2018-05-16¶
Array¶
Fix
rechunk
with chunksize of -1 in a dict (GH#3469) Stephan Hoyereinsum
now accepts thesplit_every
parameter (GH#3471) Guido Imperiale
DataFrame¶
Compatibility with pandas 0.23.0 (GH#3499) Tom Augspurger
0.17.4 / 2018-05-03¶
Dataframe¶
Add support for indexing Dask DataFrames with string subclasses (GH#3461) James Bourbeau
Allow using both sorted_index and chunksize in read_hdf (GH#3463) Pierre Bartet
Pass filesystem to arrow piece reader (GH#3466) Martin Durant
Switches to using dask.compat string_types (GH#3462) James Bourbeau
0.17.3 / 2018-05-02¶
Array¶
Add
einsum
for Dask Arrays (GH#3412) Simon PerkinsAdd
piecewise
for Dask Arrays (GH#3350) John A KirkhamFix handling of
nan
inbroadcast_shapes
(GH#3356) John A KirkhamAdd
isin
for dask arrays (GH#3363). Stephan HoyerOverhauled
topk
for Dask Arrays: faster algorithm, particularly for large k’s; added support for multiple axes, recursive aggregation, and an option to pick the bottom k elements instead. (GH#3395) Guido ImperialeThe
topk
API has changed from topk(k, array) to the more conventional topk(array, k). The legacy API still works but is now deprecated. (GH#2965) Guido ImperialeNew function
argtopk
for Dask Arrays (GH#3396) Guido ImperialeFix handling partial depth and boundary in
map_overlap
(GH#3445) John A KirkhamAdd
gradient
for Dask Arrays (GH#3434) John A Kirkham
DataFrame¶
Allow t as shorthand for table in to_hdf for pandas compatibility (GH#3330) Jörg Dietrich
Added top level isna method for Dask DataFrames (GH#3294) Christopher Ren
Fix selection on partition column on
read_parquet
forengine="pyarrow"
(GH#3207) Uwe KornAdded DataFrame.squeeze method (GH#3366) Christopher Ren
Added infer_divisions option to
read_parquet
to specify whether read engines should compute divisions (GH#3387) Jon MeaseAdded support for inferring division for
engine="pyarrow"
(GH#3387) Jon MeaseProvide more informative error message for meta= errors (GH#3343) Matthew Rocklin
add orc reader (GH#3284) Martin Durant
Default compression for parquet now always Snappy, in line with pandas (GH#3373) Martin Durant
Fixed bug in Dask DataFrame and Series comparisons with NumPy scalars (GH#3436) James Bourbeau
Remove outdated requirement from repartition docstring (GH#3440) Jörg Dietrich
Fixed bug in aggregation when only a Series is selected (GH#3446) Jörg Dietrich
Add default values to make_timeseries (GH#3421) Matthew Rocklin
Core¶
Support traversing collections in persist, visualize, and optimize (GH#3410) Jim Crist
Add schedule= keyword to compute and persist. This replaces common use of the get= keyword (GH#3448) Matthew Rocklin
0.17.2 / 2018-03-21¶
Array¶
Add
broadcast_arrays
for Dask Arrays (GH#3217) John A KirkhamAdd
bitwise_*
ufuncs (GH#3219) John A KirkhamAdd optional
axis
argument tosqueeze
(GH#3261) John A KirkhamValidate inputs to atop (GH#3307) Matthew Rocklin
Avoid calls to astype in concatenate if all parts have the same dtype (GH#3301) Martin Durant
DataFrame¶
Fixed bug in shuffle due to aggressive truncation (GH#3201) Matthew Rocklin
Support specifying categorical columns on
read_parquet
withcategories=[…]
forengine="pyarrow"
(GH#3177) Uwe KornAdd
dd.tseries.Resampler.agg
(GH#3202) Richard PostelnikSupport operations that mix dataframes and arrays (GH#3230) Matthew Rocklin
Support extra Scalar and Delayed args in
dd.groupby._Groupby.apply
(GH#3256) Gabriele Lanaro
Bag¶
Support joining against single-partitioned bags and delayed objects (GH#3254) Matthew Rocklin
Core¶
Fixed bug when using unexpected but hashable types for keys (GH#3238) Daniel Collins
Fix bug in task ordering so that we break ties consistently with the key name (GH#3271) Matthew Rocklin
Avoid sorting tasks in order when the number of tasks is very large (GH#3298) Matthew Rocklin
0.17.1 / 2018-02-22¶
Array¶
Corrected dimension chunking in indices (GH#3166, GH#3167) Simon Perkins
Inline
store_chunk
calls forstore
’sreturn_stored
option (GH#3153) John A KirkhamCompatibility with struct dtypes for NumPy 1.14.1 release (GH#3187) Matthew Rocklin
DataFrame¶
Bugfix to allow column assignment of pandas datetimes(GH#3164) Max Epstein
Core¶
New file-system for HTTP(S), allowing direct loading from specific URLs (GH#3160) Martin Durant
Fix bug when tokenizing partials with no keywords (GH#3191) Matthew Rocklin
Use more recent LZ4 API (GH#3157) Thrasibule
Introduce output stream parameter for progress bar (GH#3185) Dieter Weber
0.17.0 / 2018-02-09¶
Array¶
Added a support object-type arrays for nansum, nanmin, and nanmax (GH#3133) Keisuke Fujii
Update error handling when len is called with empty chunks (GH#3058) Xander Johnson
Fixes a metadata bug with
store
’sreturn_stored
option (GH#3064) John A KirkhamFix a bug in
optimization.fuse_slice
to properly handle when first input isNone
(GH#3076) James BourbeauSupport arrays with unknown chunk sizes in percentile (GH#3107) Matthew Rocklin
Tokenize scipy.sparse arrays and np.matrix (GH#3060) Roman Yurchak
DataFrame¶
Support month timedeltas in repartition(freq=…) (GH#3110) Matthew Rocklin
Avoid mutation in dataframe groupby tests (GH#3118) Matthew Rocklin
read_csv
,read_table
, andread_parquet
accept iterables of paths (GH#3124) Jim CristDeprecates the
dd.to_delayed
function in favor of the existing method (GH#3126) Jim CristReturn dask.arrays from df.map_partitions calls when the UDF returns a numpy array (GH#3147) Matthew Rocklin
Change handling of
columns
andindex
indd.read_parquet
to be more consistent, especially in handling of multi-indices (GH#3149) Jim Cristfastparquet append=True allowed to create new dataset (GH#3097) Martin Durant
dtype rationalization for sql queries (GH#3100) Martin Durant
Core¶
Change default task ordering to prefer nodes with few dependents and then many downstream dependencies (GH#3056) Matthew Rocklin
Add color= option to visualize to color by task order (GH#3057) (GH#3122) Matthew Rocklin
Remove short-circuit hdfs reads handling due to maintenance costs. May be re-added in a more robust manner later (GH#3079) Jim Crist
Add
dask.base.optimize
for optimizing multiple collections without computing. (GH#3071) Jim CristRename
dask.optimize
module todask.optimization
(GH#3071) Jim CristChange task ordering to do a full traversal (GH#3066) Matthew Rocklin
Adds an
optimize_graph
keyword to allto_delayed
methods to allow controlling whether optimizations occur on conversion. (GH#3126) Jim CristSupport using
pyarrow
for hdfs integration (GH#3123) Jim CristMove HDFS integration and tests into dask repo (GH#3083) Jim Crist
0.16.1 / 2018-01-09¶
Array¶
Fix handling of scalar percentile values in
percentile
(GH#3021) James BourbeauPrevent
bool()
coercion from calling compute (GH#2958) Albert DeFuscoAdd
matmul
(GH#2904) John A KirkhamSupport N-D arrays with
matmul
(GH#2909) John A KirkhamAdd
vdot
(GH#2910) John A KirkhamExplicit
chunks
argument forbroadcast_to
(GH#2943) Stephan HoyerAdd
meshgrid
(GH#2938) John A Kirkham and (GH#3001) Markus GonserPreserve singleton chunks in
fftshift
/ifftshift
(GH#2733) John A KirkhamFix handling of negative indexes in
vindex
and raise errors for out of bounds indexes (GH#2967) Stephan HoyerAdd
flip
,flipud
,fliplr
(GH#2954) John A KirkhamAdd
float_power
ufunc (GH#2962) (GH#2969) John A KirkhamCompatability for changes to structured arrays in the upcoming NumPy 1.14 release (GH#2964) Tom Augspurger
Add
block
(GH#2650) John A KirkhamAdd the
return_stored
option tostore
for chaining stored results (GH#2980) John A Kirkham
DataFrame¶
Fixed naming bug in cumulative aggregations (GH#3037) Martijn Arts
Fixed
dd.read_csv
whennames
is given butheader
is not set toNone
(GH#2976) Martijn ArtsFixed
dd.read_csv
so that passing instances ofCategoricalDtype
indtype
will result in known categoricals (GH#2997) Tom AugspurgerPrevent
bool()
coercion from calling compute (GH#2958) Albert DeFuscoDataFrame.read_sql()
(GH#2928) to an empty database tables returns an empty dask dataframe Apostolos VlachopoulosCompatability for reading Parquet files written by PyArrow 0.8.0 (GH#2973) Tom Augspurger
Correctly handle the column name (df.columns.name) when reading in
dd.read_parquet
(GH#2973) Tom AugspurgerFixed
dd.concat
losing the index dtype when the data contained a categorical (GH#2932) Tom AugspurgerDataFrame.merge()
now supports merging on a combination of columns and the index (GH#2960) Jon MeaseRemoved the deprecated
dd.rolling*
methods, in preparation for their removal in the next pandas release (GH#2995) Tom AugspurgerFix metadata inference bug in which single-partition series were mistakenly special cased (GH#3035) Jim Crist
Core¶
Improve 32-bit compatibility (GH#2937) Matthew Rocklin
Change task prioritization to avoid upwards branching (GH#3017) Matthew Rocklin
0.16.0 / 2017-11-17¶
This is a major release. It includes breaking changes, new protocols, and a large number of bug fixes.
Array¶
Add
atleast_1d
,atleast_2d
, andatleast_3d
(GH#2760) (GH#2765) John A KirkhamAdd
allclose
(GH#2771) by John A KirkhamRemove
random.different_seeds
from Dask Array API docs (GH#2772) John A KirkhamDeprecate
vnorm
in favor ofdask.array.linalg.norm
(GH#2773) John A KirkhamReimplement
unique
to be lazy (GH#2775) John A KirkhamSupport broadcasting of Dask Arrays with 0-length dimensions (GH#2784) John A Kirkham
Add
asarray
andasanyarray
to Dask Array API docs (GH#2787) James BourbeauSupport
unique
’sreturn_*
arguments (GH#2779) John A KirkhamSimplify
_unique_internal
(GH#2850) (GH#2855) John A KirkhamAvoid removing some getter calls in array optimizations (GH#2826) Jim Crist
DataFrame¶
Fixed
DataFrame.quantile
andSeries.quantile
returningnan
when missing values are present (GH#2791) Tom AugspurgerFixed
DataFrame.quantile
losing the result.name
whenq
is a scalar (GH#2791) Tom AugspurgerFixed
dd.concat
return adask.Dataframe
when concatenating a single series along the columns, matching pandas’ behavior (GH#2800) James MunroeFixed default inplace parameter for
DataFrame.eval
to match the pandas defualt for pandas >= 0.21.0 (GH#2838) Tom AugspurgerFix exception when calling
DataFrame.set_index
on text column where one of the partitions was empty (GH#2831) Jesse VogtDo not raise exception when calling
DataFrame.set_index
on empty dataframe (GH#2827) Jesse VogtFixed bug in
Dataframe.fillna
when filling with aSeries
value (GH#2810) Tom AugspurgerDeprecate old argument ordering in
dd.to_parquet
to better match convention of putting the dataframe first (GH#2867) Jim Cristdf.astype(categorical_dtype -> known categoricals (GH#2835) Jim Crist
Test against Pandas release candidate (GH#2814) Tom Augspurger
Add more tests for read_parquet(engine=’pyarrow’) (GH#2822) Uwe Korn
Remove unnecessary map_partitions in aggregate (GH#2712) Christopher Prohm
Fix bug calling sample on empty partitions (GH#2818) @xwang777
Error nicely when parsing dates in read_csv (GH#2863) Jim Crist
Cleanup handling of passing filesystem objects to PyArrow readers (GH#2527) @fjetter
Support repartitioning even if there are no divisions (GH#2873) @Ced4
Support reading/writing to hdfs using
pyarrow
indd.to_parquet
(GH#2894, GH#2881) Jim Crist
Core¶
Allow tuples as sharedict keys (GH#2763) Matthew Rocklin
Calling compute within a dask.distributed task defaults to distributed scheduler (GH#2762) Matthew Rocklin
Auto-import gcsfs when gcs:// protocol is used (GH#2776) Matthew Rocklin
Fully remove dask.async module, use dask.local instead (GH#2828) Thomas Caswell
Compatability with bokeh 0.12.10 (GH#2844) Tom Augspurger
Update Dask collection interface during XArray integration (GH#2847) Matthew Rocklin
Close resource profiler process on __exit__ (GH#2871) Jim Crist
Fix port for bokeh dashboard in docs (GH#2889) Ian Hopkinson
Wrap Dask filesystems for PyArrow compatibility (GH#2881) Jim Crist
0.15.3 / 2017-09-24¶
Array¶
Add masked arrays (GH#2301)
Add
*_like array creation functions
(GH#2640)Indexing with unsigned integer array (GH#2647)
Improved slicing with boolean arrays of different dimensions (GH#2658)
Support literals in
top
andatop
(GH#2661)Optional axis argument in cumulative functions (GH#2664)
Improve tests on scalars with
assert_eq
(GH#2681)Fix norm keepdims (GH#2683)
Add
ptp
(GH#2691)Add apply_along_axis (GH#2690) and apply_over_axes (GH#2702)
DataFrame¶
Added
Series.str[index]
(GH#2634)Allow the groupby by param to handle columns and index levels (GH#2636)
DataFrame.to_csv
andBag.to_textfiles
now return the filenames towhich they have written (GH#2655)
Fix combination of
partition_on
andappend
into_parquet
(GH#2645)Fix for parquet file schemes (GH#2667)
Repartition works with mixed categoricals (GH#2676)
0.15.2 / 2017-08-25¶
Array¶
Remove spurious keys from map_overlap graph (GH#2520)
where works with non-bool condition and scalar values (GH#2543) (GH#2549)
Add argwhere, _nonzero, and where(cond) (GH#2539)
Generalize vindex in dask.array to handle multi-dimensional indices (GH#2573)
Add choose method (GH#2584)
Split code into reorganized files (GH#2595)
Add linalg.norm (GH#2597)
Improve dtype inference and reflection (GH#2571)
DataFrame¶
0.15.1 / 2017-07-08¶
0.15.0 / 2017-06-09¶
Array¶
Add dask.array.stats submodule (GH#2269)
Support
ufunc.outer
(GH#2345)Optimize fancy indexing by reducing graph overhead (GH#2333) (GH#2394)
Faster array tokenization using alternative hashes (GH#2377)
Added the matmul
@
operator (GH#2349)Improved coverage of the
numpy.fft
module (GH#2320) (GH#2322) (GH#2327) (GH#2323)Support NumPy’s
__array_ufunc__
protocol (GH#2438)
Bag¶
0.14.2 / 2017-05-03¶
Array¶
Add da.indices (GH#2268), da.tile (GH#2153), da.roll (GH#2135)
Simultaneously support drop_axis and new_axis in da.map_blocks (GH#2264)
Rechunk and concatenate work with unknown chunksizes (GH#2235) and (GH#2251)
Support non-numpy container arrays, notably sparse arrays (GH#2234)
Tensordot contracts over multiple axes (GH#2186)
Allow delayed targets in da.store (GH#2181)
Support interactions against lists and tuples (GH#2148)
Constructor plugins for debugging (GH#2142)
Multi-dimensional FFTs (single chunk) (GH#2116)
DataFrame¶
0.14.1 / 2017-03-22¶
Array¶
Micro-optimize optimizations (GH#2058)
Change slicing optimizations to avoid fusing raw numpy arrays (GH#2075) (GH#2080)
Dask.array operations now work on numpy arrays (GH#2079)
Reshape now works in a much broader set of cases (GH#2089)
Support deepcopy python protocol (GH#2090)
Allow user-provided FFT implementations in
da.fft
(GH#2093)
DataFrame¶
Fix to_parquet with empty partitions (GH#2020)
Optional
npartitions='auto'
mode inset_index
(GH#2025)Optimize shuffle performance (GH#2032)
Support efficient repartitioning along time windows like
repartition(freq='12h')
(GH#2059)Improve speed of categorize (GH#2010)
Support single-row dataframe arithmetic (GH#2085)
Automatically avoid shuffle when setting index with a sorted column (GH#2091)
Improve handling of integer-na handling in read_csv (GH#2098)
0.14.0 / 2017-02-24¶
Array¶
Bag¶
DataFrame¶
Delayed¶
Core¶
Improve windows path parsing in corner cases (GH#1910)
Rename tasks when fusing (GH#1919)
Add top level
persist
function (GH#1927)Propagate
errors=
keyword in byte handling (GH#1954)Dask.compute traverses Python collections (GH#1975)
Structural sharing between graphs in dask.array and dask.delayed (GH#1985)
0.13.0 / 2017-01-02¶
Array¶
Mandatory dtypes on dask.array. All operations maintain dtype information and UDF functions like map_blocks now require a dtype= keyword if it can not be inferred. (GH#1755)
Support arrays without known shapes, such as arises when slicing arrays with arrays or converting dataframes to arrays (GH#1838)
Support mutation by setting one array with another (GH#1840)
Tree reductions for covariance and correlations. (GH#1758)
Add SerializableLock for better use with distributed scheduling (GH#1766)
Improved atop support (GH#1800)
DataFrame¶
Add
map_overlap
for custom rolling operations (GH#1769)Add
shift
(GH#1773)Add Parquet support (GH#1782) (GH#1792) (GH#1810), (GH#1843), (GH#1859), (GH#1863)
Add missing methods combine, abs, autocorr, sem, nsmallest, first, last, prod, (GH#1787)
Reductions with multiple output partitions (for operations like drop_duplicates) (GH#1808), (GH#1823) (GH#1828)
Add delitem and copy to DataFrames, increasing mutation support (GH#1858)
Delayed¶
Changed behaviour for
delayed(nout=0)
anddelayed(nout=1)
:delayed(nout=1)
does not default toout=None
anymore, anddelayed(nout=0)
is also enabled. I.e. functions with return tuples of length 1 or 0 can be handled correctly. This is especially handy, if functions with a variable amount of outputs are wrapped bydelayed
. E.g. a trivial example:delayed(lambda *args: args, nout=len(vals))(*vals)
0.12.0 / 2016-11-03¶
DataFrame¶
Return a series when functions given to
dataframe.map_partitions
return scalars (GH#1515)Fix type size inference for series (GH#1513)
dataframe.DataFrame.categorize
no longer includes missing values in thecategories
. This is for compatibility with a pandas change (GH#1565)Fix head parser error in
dataframe.read_csv
when some lines have quotes (GH#1495)Add
dataframe.reduction
andseries.reduction
methods to apply generic row-wise reduction to dataframes and series (GH#1483)Add
dataframe.select_dtypes
, which mirrors the pandas method (GH#1556)dataframe.read_hdf
now supports readingSeries
(GH#1564)Support Pandas 0.19.0 (GH#1540)
Implement
select_dtypes
(GH#1556)String accessor works with indexes (GH#1561)
Add pipe method to dask.dataframe (GH#1567)
Add
indicator
keyword to merge (GH#1575)Support Series in
read_hdf
(GH#1575)Support Categories with missing values (GH#1578)
Support inplace operators like
df.x += 1
(GH#1585)Str accessor passes through args and kwargs (GH#1621)
Improved groupby support for single-machine multiprocessing scheduler (GH#1625)
Tree reductions (GH#1663)
Pivot tables (GH#1665)
Add clip (GH#1667), align (GH#1668), combine_first (GH#1725), and any/all (GH#1724)
Improved handling of divisions on dask-pandas merges (GH#1666)
Add
groupby.aggregate
method (GH#1678)Add
dd.read_table
function (GH#1682)Support 2d indexing in
loc
(GH#1726)Extend
resample
to include DataFrames (GH#1741)Support dask.array ufuncs on dask.dataframe objects (GH#1669)
Array¶
Add information about how
dask.array
chunks
argument work (GH#1504)Fix field access with non-scalar fields in
dask.array
(GH#1484)Add concatenate= keyword to atop to concatenate chunks of contracted dimensions
Extend
atop
with aconcatenate=
(GH#1609)new_axes=
(GH#1612) andadjust_chunks=
(GH#1716) keywordsAdd clip (GH#1610) swapaxes (GH#1611) round (GH#1708) repeat
Automatically align chunks in
atop
-backed operations (GH#1644)Cull dask.arrays on slicing (GH#1709)
Bag¶
Administration¶
Added changelog (GH#1526)
Create new threadpool when operating from thread (GH#1487)
Unify example documentation pages into one (GH#1520)
Add versioneer for git-commit based versions (GH#1569)
Pass through node_attr and edge_attr keywords in dot visualization (GH#1614)
Add continuous testing for Windows with Appveyor (GH#1648)
Remove use of multiprocessing.Manager (GH#1653)
Add global optimizations keyword to compute (GH#1675)
Micro-optimize get_dependencies (GH#1722)
0.11.0 / 2016-08-24¶
Major Points¶
DataFrames now enforce knowing full metadata (columns, dtypes) everywhere.
Previously we would operate in an ambiguous state when functions lost dtype
information (such as apply
). Now all dataframes always know their dtypes
and raise errors asking for information if they are unable to infer (which
they usually can). Some internal attributes like _pd
and
_pd_nonempty
have been moved.
The internals of the distributed scheduler have been refactored to transition tasks between explicit states. This improves resilience, reasoning about scheduling, plugin operation, and logging. It also makes the scheduler code easier to understand for newcomers.
Breaking Changes¶
The
distributed.s3
anddistributed.hdfs
namespaces are gone. Use protocols in normal methods likeread_text('s3://...'
instead.Dask.array.reshape
now errs in some cases where previously it would have create a very large number of tasks
0.10.2 / 2016-07-27¶
More Dataframe shuffles now work in distributed settings, ranging from setting-index to hash joins, to sorted joins and groupbys.
Dask passes the full test suite when run when under in Python’s optimized-OO mode.
On-disk shuffles were found to produce wrong results in some highly-concurrent situations, especially on Windows. This has been resolved by a fix to the partd library.
Fixed a growth of open file descriptors that occurred under large data communications
Support ports in the
--bokeh-whitelist
option ot dask-scheduler to better routing of web interface messages behind non-trivial network settingsSome improvements to resilience to worker failure (though other known failures persist)
You can now start an IPython kernel on any worker for improved debugging and analysis
Improvements to
dask.dataframe.read_hdf
, especially when reading from multiple files and docs
0.10.0 / 2016-06-13¶
Major Changes¶
This version drops support for Python 2.6
Conda packages are built and served from conda-forge
The
dask.distributed
executables have been renamed from dfoo to dask-foo. For example dscheduler is renamed to dask-schedulerBoth Bag and DataFrame include a preliminary distributed shuffle.
Bag¶
Add task-based shuffle for distributed groupbys
Add accumulate for cumulative reductions
DataFrame¶
Add a task-based shuffle suitable for distributed joins, groupby-applys, and set_index operations. The single-machine shuffle remains untouched (and much more efficient.)
Add support for new Pandas rolling API with improved communication performance on distributed systems.
Add
groupby.std/var
Pass through S3/HDFS storage options in
read_csv
Improve categorical partitioning
Add eval, info, isnull, notnull for dataframes
Distributed¶
Rename executables like dscheduler to dask-scheduler
Improve scheduler performance in the many-fast-tasks case (important for shuffling)
Improve work stealing to be aware of expected function run-times and data sizes. The drastically increases the breadth of algorithms that can be efficiently run on the distributed scheduler without significant user expertise.
Support maximum buffer sizes in streaming queues
Improve Windows support when using the Bokeh diagnostic web interface
Support compression of very-large-bytestrings in protocol
Support clean cancellation of submitted futures in Joblib interface
Other¶
All dask-related projects (dask, distributed, s3fs, hdfs, partd) are now building conda packages on conda-forge.
Change credential handling in s3fs to only pass around delegated credentials if explicitly given secret/key. The default now is to rely on managed environments. This can be changed back by explicitly providing a keyword argument. Anonymous mode must be explicitly declared if desired.
0.9.0 / 2016-05-11¶
API Changes¶
dask.do
anddask.value
have been renamed todask.delayed
dask.bag.from_filenames
has been renamed todask.bag.read_text
All S3/HDFS data ingest functions like
db.from_s3
ordistributed.s3.read_csv
have been moved into the plainread_text
,read_csv functions
, which now support protocols, likedd.read_csv('s3://bucket/keys*.csv')
Array¶
Add support for
scipy.LinearOperator
Improve optional locking to on-disk data structures
Change rechunk to expose the intermediate chunks
Bag¶
Rename
from_filename
s toread_text
Remove
from_s3
in favor ofread_text('s3://...')
DataFrame¶
Fixed numerical stability issue for correlation and covariance
Allow no-hash
from_pandas
for speedy round-trips to and from-pandas objectsGenerally reengineered
read_csv
to be more in line with Pandas behaviorSupport fast
set_index
operations for sorted columns
Delayed¶
Rename
do/value
todelayed
Rename
to/from_imperative
toto/from_delayed
Distributed¶
Move s3 and hdfs functionality into the dask repository
Adaptively oversubscribe workers for very fast tasks
Improve PyPy support
Improve work stealing for unbalanced workers
Scatter data efficiently with tree-scatters
Other¶
Add lzma/xz compression support
Raise a warning when trying to split unsplittable compression types, like gzip or bz2
Improve hashing for single-machine shuffle operations
Add new callback method for start state
General performance tuning
0.8.1 / 2016-03-11¶
Array¶
Bugfix for range slicing that could periodically lead to incorrect results.
Improved support and resiliency of
arg
reductions (argmin
,argmax
, etc.)
Bag¶
Add
zip
function
DataFrame¶
Add
corr
andcov
functionsAdd
melt
functionBugfixes for io to bcolz and hdf5
0.8.0 / 2016-02-20¶
Array¶
Changed default array reduction split from 32 to 4
Linear algebra,
tril
,triu
,LU
,inv
,cholesky
,solve
,solve_triangular
,eye
,lstsq
,diag
,corrcoef
.
Bag¶
Add tree reductions
Add range function
drop
from_hdfs
function (better functionality now exists in hdfs3 and distributed projects)
DataFrame¶
Refactor
dask.dataframe
to include a full empty pandas dataframe as metadata. Drop the.columns
attribute on SeriesAdd Series categorical accessor, series.nunique, drop the
.columns
attribute for series.read_csv
fixes (multi-column parse_dates, integer column names, etc. )Internal changes to improve graph serialization
Other¶
Documentation updates
Add from_imperative and to_imperative functions for all collections
Aesthetic changes to profiler plots
Moved the dask project to a new dask organization
0.7.6 / 2016-01-05¶
Array¶
Improve thread safety
Tree reductions
Add
view
,compress
,hstack
,dstack
,vstack
methodsmap_blocks
can now remove and add dimensions
DataFrame¶
Improve thread safety
Extend sampling to include replacement options
Imperative¶
Removed optimization passes that fused results.
Core¶
Removed
dask.distributed
Improved performance of blocked file reading
Serialization improvements
Test Python 3.5
0.7.4 / 2015-10-23¶
This was mostly a bugfix release. Some notable changes:
Fix minor bugs associated with the release of numpy 1.10 and pandas 0.17
Fixed a bug with random number generation that would cause repeated blocks due to the birthday paradox
Use locks in
dask.dataframe.read_hdf
by default to avoid concurrency issuesChange
dask.get
to point todask.async.get_sync
by defaultAllow visualization functions to accept general graphviz graph options like rankdir=’LR’
Add reshape and ravel to
dask.array
Support the creation of
dask.arrays
fromdask.imperative
objects
Deprecation¶
This release also includes a deprecation warning for dask.distributed
, which
will be removed in the next version.
Future development in distributed computing for dask is happening here: https://distributed.dask.org . General feedback on that project is most welcome from this community.
0.7.3 / 2015-09-25¶
Diagnostics¶
A utility for profiling memory and cpu usage has been added to the
dask.diagnostics
module.
DataFrame¶
This release improves coverage of the pandas API. Among other things
it includes nunique
, nlargest
, quantile
. Fixes encoding issues
with reading non-ascii csv files. Performance improvements and bug fixes
with resample. More flexible read_hdf with globbing. And many more. Various
bug fixes in dask.imperative
and dask.bag
.
0.7.0 / 2015-08-15¶
DataFrame¶
This release includes significant bugfixes and alignment with the Pandas API. This has resulted both from use and from recent involvement by Pandas core developers.
New operations: query, rolling operations, drop
Improved operations: quantiles, arithmetic on full dataframes, dropna, constructor logic, merge/join, elemwise operations, groupby aggregations
Bag¶
Fixed a bug in fold where with a null default argument
Array¶
New operations: da.fft module, da.image.imread
Infrastructure¶
The array and dataframe collections create graphs with deterministic keys. These tend to be longer (hash strings) but should be consistent between computations. This will be useful for caching in the future.
All collections (Array, Bag, DataFrame) inherit from common subclass
0.6.1 / 2015-07-23¶
Distributed¶
Improved (though not yet sufficient) resiliency for
dask.distributed
when workers die
DataFrame¶
Improved writing to various formats, including to_hdf, to_castra, and to_csv
Improved creation of dask DataFrames from dask Arrays and Bags
Improved support for categoricals and various other methods
Array¶
Various bug fixes
Histogram function
Scheduling¶
Added tie-breaking ordering of tasks within parallel workloads to better handle and clear intermediate results
Other¶
Added the dask.do function for explicit construction of graphs with normal python code
Traded pydot for graphviz library for graph printing to support Python3
There is also a gitter chat room and a stackoverflow tag