gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
5039 commits 28 branches 278 tags 135 MiB
Commit graph

3188 commits

Author SHA1 Message Date
Chris Lu 0ef43a23a7 go fmt 2021-01-12 02:30:12 -08:00
Chris Lu 629c9962e7 filer: add leveldb3 
support dedicated leveldb instance for each bucket
 2021-01-12 02:29:44 -08:00
Chris Lu 361043e6c1 filer store: leveldb2 fix nil entry error if not found 2021-01-12 02:28:57 -08:00
Chris Lu cfb9342a15 avoid concurrent map updates to viper 2021-01-12 02:28:13 -08:00
Chris Lu 38d516251e Revert "mount: avoid fatal error" 
This reverts commit a4f0bac9b7.
 2021-01-11 13:50:15 -08:00
Chris Lu a4f0bac9b7 mount: avoid fatal error 2021-01-11 13:26:59 -08:00
Chris Lu 8a7302ce37 s3: KeyCount is zero 
fix https://github.com/chrislusf/seaweedfs/issues/1741
 2021-01-11 03:50:45 -08:00
Chris Lu 2c7148cd55 redis cluster: do not enable route by latency by default 2021-01-11 02:43:53 -08:00
Chris Lu acc0d5cd2c Revert "Revert "filer store: redis module upgrade"" 
This reverts commit dd0171e35c.
 2021-01-11 02:30:19 -08:00
Chris Lu dd0171e35c Revert "filer store: redis module upgrade" 
This reverts commit 40161c9cec.
 2021-01-11 02:09:18 -08:00
Chris Lu 394513f598 filer: ensure seamless meta data updates 2021-01-11 02:08:55 -08:00
Chris Lu 6e12a3a490 refactoring 2021-01-11 02:08:26 -08:00
Chris Lu 62ee484d12 refactoring 2021-01-11 00:03:13 -08:00
Chris Lu 1efb51ba84 filer: change to saveToFilerLimit from cacheToFilerLimit 
short circuit saving small files to volume server
 2021-01-10 23:14:46 -08:00
Chris Lu f0d3b3bf93 2.20 2021-01-09 22:55:22 -08:00
Chris Lu d3d3f2fb9b mount: default to 128 concurrent writers 2021-01-09 22:53:37 -08:00
Chris Lu 90b117acf1 update ccache version 2021-01-08 02:17:43 -08:00
Chris Lu e2e0e30a51 filer: redis default to false for useReadOnly and routeByLatency 2021-01-08 01:12:44 -08:00
Chris Lu 40161c9cec filer store: redis module upgrade 2021-01-08 01:08:44 -08:00
Chris Lu 30efaa66c6 server: add option to disable master 2021-01-07 02:35:47 -08:00
Chris Lu 07f4703bfc
Merge pull request #1738 from qieqieplus/rocksdb 
fix rocksdb crash when list directory
 2021-01-07 02:26:41 -08:00
qieqieplus 0f18592315 alter style 2021-01-07 18:21:48 +08:00
qieqieplus c7e7431fd3 fix seek lastkey may reach EOF 2021-01-07 17:07:56 +08:00
Chris Lu e327385644 go fmt 2021-01-06 04:22:00 -08:00
Chris Lu 2b76854641 add "weed filer.cat" to read files directly from volume servers 2021-01-06 04:22:00 -08:00
Chris Lu 95ecf0c72f
Merge pull request #1735 from qieqieplus/rocksdb 
ignore decode error for non-entry data
 2021-01-06 02:22:18 -08:00
qieqieplus 0764fccde7 ignore decode error for non-entry data 2021-01-06 17:42:43 +08:00
Chris Lu ab57767d4b
Merge pull request #1730 from qieqieplus/rocksdb 
impl: TTL per entry for rocksdb; fix package name
 2021-01-05 13:24:45 -08:00
qieqieplus b2e50f602f
fix time comparison 2021-01-05 19:36:40 +08:00
qieqieplus bbae13e415 impl: TTL per entry for rocksdb; fix package name 2021-01-05 17:58:40 +08:00
henry b42c5b71c1 Seaweed custom header are not visible to Vue or javascript 2021-01-05 13:01:29 +08:00
henry 1309dfe612 Seaweed custom header are not visible to Vue or javascript 2021-01-05 12:58:46 +08:00
henry 942a30f04c remove sort 2021-01-05 12:24:32 +08:00
henry a3c432dadc Merge branch 'master' of https://github.com/fuyouyshengwu/seaweedfs 2021-01-05 11:17:29 +08:00
qieqieplus ebb223c190 fix for rocksdb kv 2021-01-04 21:21:46 +08:00
chenqieqie 40b3207f18 fix #1726 2021-01-04 18:48:55 +08:00
henry 97a94eddab Merge branch 'master' of https://github.com/fuyouyshengwu/seaweedfs 2021-01-04 14:50:01 +08:00
Chris Lu 2ce86f308e 2.19 2021-01-03 17:24:02 -08:00
Chris Lu 9c9ba3c209 nil related 
related to https://github.com/chrislusf/seaweedfs/issues/1676
 2021-01-03 12:25:58 -08:00
Chris Lu 4b5adc60b4 upload result changes to the right output variable 2021-01-03 01:44:22 -08:00
Chris Lu 039ad101b1 manifest file also need to detect compression 
fix https://github.com/chrislusf/seaweedfs/issues/1724#issuecomment-753585046
 2021-01-03 01:42:21 -08:00
Chris Lu 5f538c5f9d Merge branch 'add_rocksdb' 2021-01-03 00:48:56 -08:00
Chris Lu bdd166c031 adjust readme 2021-01-03 00:47:58 -08:00
Chris Lu a04d8927a8 add scaffold 2021-01-03 00:44:52 -08:00
Chris Lu 4250a3a675 upload: add read default collection from master 
fix https://github.com/chrislusf/seaweedfs/issues/1724#issuecomment-753576071
 2021-01-02 23:43:01 -08:00
Chris Lu f4d87f8b9d works 
performance seems about the same as leveldb for insert entries
 2021-01-02 23:32:58 -08:00
Chris Lu 2c3c2c27d7 separate prefix from namePattern 
fix https://github.com/chrislusf/seaweedfs/issues/1722
 2021-01-01 20:23:23 -08:00
Peter Cai cd2a9865ed filer: respect Content-Type on PUT 
This is needed for the S3 gateway to conform to the S3 protocol -- the
server should always respect the Content-Type set by client.
 2020-12-31 11:32:45 +08:00
Chris Lu 3433accb1b weed upload: add TTL 2020-12-30 15:46:07 -08:00
henry 14ddd15508 1 Distinguish between folders and common files, folders are listed first 
2 Add some custom headers that need to be exposed
 2020-12-31 00:03:22 +08:00
Chris Lu adef4ddc87 mount: load security keys 
fix https://github.com/chrislusf/seaweedfs/issues/1714
 2020-12-29 11:51:38 -08:00
Chris Lu 89977123b6 shell: add shell.toml support 
fix https://github.com/chrislusf/seaweedfs/issues/1664
 2020-12-28 15:07:18 -08:00
Chris Lu d2ead72918 2.18 2020-12-28 12:37:10 -08:00
Chris Lu 2e6cf541e7 filer: avoid deletion error when item is not found 2020-12-28 00:18:37 -08:00
Chris Lu 6d2c979999 s3: fix regression 
fix https://github.com/chrislusf/seaweedfs/issues/1707
 2020-12-27 21:09:45 -08:00
Chris Lu 90df52d001 s3: add command option "allowEmptyFolder" to filer and server 2020-12-27 21:09:06 -08:00
Chris Lu da7e5aaa65 filer: optimize for less number of directory lookup 
bottom up directory lookup
 2020-12-27 20:46:17 -08:00
henry 7458ff9523 Solve the Chinese name disorder of download file 2020-12-28 11:28:33 +08:00
Chris Lu 561a2ff0bc s3: add option for "alllowEmptyFolder" 2020-12-27 10:50:27 -08:00
Chris Lu aa020ee7e7 s3: restore V2 auth type 
The added headers were included in calculating the signatures, failing the comparison
 2020-12-26 22:01:16 -08:00
Chris Lu da87f6b265 remove unused code 2020-12-26 15:21:12 -08:00
Chris Lu 0a7c5f85a9 filer: add namePattern to search in current folder 2020-12-26 15:05:31 -08:00
Chris Lu 515b7632d7 2.17 2020-12-25 23:58:22 -08:00
Chris Lu 0a067944cc filer: add retries during volume moving 
fix https://github.com/chrislusf/seaweedfs/issues/1704
 2020-12-25 02:32:55 -08:00
Chris Lu 090f85be4b s3: support config action Admin:bucket 2020-12-25 00:38:56 -08:00
Chris Lu 75613b2cc7 Merge branch 'add_hbase' 2020-12-24 12:13:05 -08:00
Chris Lu 0e016bc7bd hbase add ttl 2020-12-24 12:10:35 -08:00
Chris Lu 8e48a235e2 s3: avoid looping if the directory is empty 
fix https://github.com/chrislusf/seaweedfs/issues/1701
 2020-12-24 11:34:52 -08:00
Chris Lu 1620de08ae added more logs 2020-12-24 01:52:06 -08:00
Chris Lu a09cd28986 add debug messages 2020-12-24 01:42:15 -08:00
Chris Lu 75c6edba9e filer: hbase add enabled flag 2020-12-24 00:19:16 -08:00
Chris Lu 9bf6c10505 fix prefix 2020-12-24 00:18:59 -08:00
Chris Lu 2fea8cfa0f fix compilation 2020-12-23 23:54:15 -08:00
Chris Lu 53bc1ea25b fix compilation 2020-12-23 23:53:46 -08:00
Chris Lu b5e2be635a adjust for directory listing 2020-12-23 23:49:22 -08:00
Chris Lu c4a202ec41 fix wrong column family 2020-12-23 23:23:05 -08:00
Chris Lu c3d1b3b5aa hook up 2020-12-23 21:49:01 -08:00
Chris Lu 94e3757c08 fix 2020-12-23 21:45:16 -08:00
Chris Lu 64c48c9724 seems compiling 2020-12-23 21:44:53 -08:00
Chris Lu 3be3635799 just log error when checking empty dir failed 2020-12-23 14:36:48 -08:00
Chris Lu 0ca9d89589 s3: break loop if error 
fix #1701
 2020-12-23 14:34:59 -08:00
Baptiste Mille-Mathias 385a4dec30 [shell] Clarify bucket replication setting order 2020-12-23 09:35:24 +01:00
Chris Lu da134a2eb7 minor 2020-12-22 17:43:13 -08:00
Chris Lu beb3b8ddb0 shell: change bucket.list to s3.bucket.list, same for create and delete 2020-12-22 17:40:55 -08:00
Chris Lu 4cf3176cbb filer first, master second 2020-12-22 17:34:44 -08:00
Chris Lu a1009e8044 filer: add -defaultStoreDir so that filer.toml can be skipped 
fix https://github.com/chrislusf/seaweedfs/issues/1659
 2020-12-22 17:33:40 -08:00
Chris Lu a0990b929d minor 2020-12-22 17:33:40 -08:00
Baptiste Mille-Mathias 0ed5345954 [shell] Add more information to bucket.create 
State the replication setting that will be honor if not defined.
 2020-12-22 17:18:59 +01:00
Chris Lu 97e3432dfe avoid wrong error 
fix https://github.com/chrislusf/seaweedfs/issues/1691
 2020-12-22 02:46:24 -08:00
Chris Lu 4f31c1bb94 go fmt 2020-12-22 02:34:08 -08:00
Chris Lu 6c4f32d173 remove unused code 2020-12-22 02:33:31 -08:00
Chris Lu 90d785a15f filer: redis, redis cluster, cassandra support super large directory 2020-12-22 02:26:05 -08:00
Chris Lu fe46411cd4 handle directory listing edge cases 2020-12-21 23:37:43 -08:00
Chris Lu c74bede730 handle listing directories 2020-12-21 23:19:05 -08:00
Chris Lu adf8cb4000 filer: path-specific stores trim out common prefixes 2020-12-21 22:57:13 -08:00
Chris Lu 0823bde8d8 refactor: split into two files 2020-12-21 22:05:15 -08:00
Chris Lu 8ab70ccf76 refactor 2020-12-21 21:46:32 -08:00
Chris Lu 65cc6dc636 refactor to dedicated function 2020-12-21 21:18:34 -08:00
Chris Lu 488c2680e8 change default filer store root 2020-12-21 21:15:22 -08:00
Chris Lu f5a19a87be filer: report error if the path specific store is not found 
related https://github.com/chrislusf/seaweedfs/issues/1691
 2020-12-21 12:43:52 -08:00
Chris Lu 6912bf94ae 2.16 2020-12-20 15:30:12 -08:00
Chris Lu 41c0f3ad24 filer: support path-specific filer store 2020-12-19 01:27:09 -08:00
Chris Lu 0d5683fb0e todo: load path-specific store from filer.toml 2020-12-18 03:05:42 -08:00
Chris Lu 3269fd7eaf filer: use store by path 2020-12-18 02:57:49 -08:00
Chris Lu e605f1e001 refactor, add getActualStore() by path 2020-12-18 02:55:00 -08:00
Chris Lu 23903aa95b refactoring 2020-12-18 02:35:45 -08:00
Chris Lu 4e6b316913 assert FilerStoreWrapper is VirtualFilerStore 2020-12-18 02:17:06 -08:00
Chris Lu 738c4fd203 fix logging when error 
address https://github.com/chrislusf/seaweedfs/pull/1685#issuecomment-747845601
 2020-12-17 19:37:46 -08:00
Chris Lu e2076201d7 volume: avoid reprocessing the same volume 
fix https://github.com/chrislusf/seaweedfs/issues/1682
 2020-12-17 13:03:39 -08:00
Chris Lu 986cbdf7d9 Revert "Merge pull request #1683 from qieqieplus/master" 
This reverts commit 8cb67952db, reversing
changes made to 200e56215a.
 2020-12-17 12:46:42 -08:00
Chris Lu b789767cd1 filer: return http status 499 for client closed connection 
fix https://github.com/chrislusf/seaweedfs/issues/1684
 2020-12-16 08:18:00 -08:00
chenqieqie 45f902a9b7 smaller critical section 2020-12-16 20:23:51 +08:00
chenqieqie 4e58a4f24e fix race condition when loading volumes concurrently 2020-12-16 18:49:10 +08:00
Chris Lu 3c7f9633eb show error a little bit more obviously 
fix https://github.com/chrislusf/seaweedfs/issues/1678
 2020-12-14 09:39:33 -08:00
Chris Lu 3d47c38262 collection.delete requires _default_ as the default empty collection name 
fix https://github.com/chrislusf/seaweedfs/issues/1677
 2020-12-14 01:05:20 -08:00
Chris Lu b7e3ca9172
Merge pull request #1650 from kmlebedev/masterServerEnableUI 
security master use access.ui
 2020-12-14 00:31:51 -08:00
Chris Lu 021358749b better detect IP address 
following advice from https://github.com/chrislusf/seaweedfs/issues/1671
 2020-12-14 00:30:20 -08:00
Konstantin Lebedev c4459249ef ui interface is always on 2020-12-14 13:24:41 +05:00
Chris Lu 38fc89d041 filer store: sql update if any insert error happens 
fix https://github.com/chrislusf/seaweedfs/issues/1673
 2020-12-13 20:49:44 -08:00
Chris Lu 23280257df sql put kv: avoid unnecessary update 2020-12-13 19:45:47 -08:00
Chris Lu 813453f3e1 2.15 2020-12-13 03:45:35 -08:00
Chris Lu 16cd6fb278 fix variable name 2020-12-12 21:19:40 -08:00
Chris Lu 5c465293e9 correctly determine whether a folder is empty 
avoid edge cases that deleting the folder if previous 32 directories are all empty

early terminate if one file is found
 2020-12-12 16:19:29 -08:00
Chris Lu 14910d035c the pagination size was too big for recursive deletion 2020-12-12 13:26:10 -08:00
Chris Lu f930c713fc more efficient recursion 2020-12-12 13:25:19 -08:00
Chris Lu 316d1b4e69 refactor APIs 2020-12-12 12:42:53 -08:00
Chris Lu 03637d6f57 s3: move "delete-directory-if-empty" to read time 
move "delete-directory-if-empty" to read time instead of entry deletion time

the listing speed for a s3 bucket folder will slow down if it has many sub folders

related to 0d345ac97d

fix https://github.com/chrislusf/seaweedfs/issues/1647

fix https://github.com/chrislusf/seaweedfs/issues/1670
 2020-12-12 03:38:34 -08:00
Chris Lu 37075a414d adjust logs 2020-12-12 03:33:57 -08:00
Chris Lu ae655033ac adjust logging 2020-12-11 16:57:53 -08:00
Chris Lu d986c7196d use append time instead of filer's own modification time 
fix https://github.com/chrislusf/seaweedfs/issues/1669
 2020-12-11 16:55:18 -08:00
Chris Lu 3fedfec1e7 check cross device rename error 2020-12-10 23:50:32 -08:00
Chris Lu c2f18a10cb minor 2020-12-10 22:23:22 -08:00
Chris Lu 4b0c2a846b add logs 2020-12-10 19:55:28 -08:00
Chris Lu 83078ac6ce filer: change to /etc/seaweedfs folder on filer 
fix https://github.com/chrislusf/seaweedfs/issues/1666
 2020-12-10 11:11:02 -08:00
Chris Lu 26731694f8 s3: use static configuration by default 
So that users can still use the previous configuration files.

If leave it empty, s3 will try to use the version from filer
 2020-12-10 00:59:04 -08:00
Chris Lu 765b3ef1be save /etc/iam/identity.json inside filer store 2020-12-10 00:15:22 -08:00
Chris Lu 97c942b5ee paginate through large directories 2020-12-09 23:23:38 -08:00
Chris Lu 02a160c3fd s3: recursively iterate all sub folders 
fix https://github.com/chrislusf/seaweedfs/issues/1656
 2020-12-09 16:47:34 -08:00
Chris Lu a9c619c270 added error handling for saving 2020-12-09 01:34:24 -08:00
Chris Lu 4211601eab set file handle reader to nil 2020-12-08 22:48:18 -08:00
Chris Lu b52ae9cef8 add s3 debug 2020-12-08 22:27:04 -08:00
Chris Lu 8e78187a97 add back last read chunk cache to reader and properly close the reader 2020-12-08 22:26:46 -08:00
Chris Lu 900d22c6ec mount: avoid memory leaking read buffer 
fix https://github.com/chrislusf/seaweedfs/issues/1654

the reader goes together with the file handle, which may stay for a long time.
 2020-12-08 02:38:53 -08:00
Chris Lu 06bb7bf6c0 adding /usr/local/etc/seaweedfs for bsd style config search path 
related to https://reviews.freebsd.org/D27391
 2020-12-07 16:46:48 -08:00
Chris Lu fce8803087 break import cycle 2020-12-07 00:29:17 -08:00
Chris Lu 606051cc79 filer: cache "/etc" filder small files to filer store 2020-12-07 00:11:35 -08:00
Chris Lu ac22f1dd26 filer subscribe: handle rename subscription 2020-12-07 00:10:49 -08:00
Chris Lu eed87791b7 s3: subscribe to s3.configure changes 2020-12-07 00:10:29 -08:00
Chris Lu 01e2da5782 refactoring 2020-12-06 23:16:20 -08:00
Chris Lu 77286f8bea fix test 2020-12-06 21:56:13 -08:00